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■ PREFACE 



Since the Job Training Partnership Act (JTPA) was first enacted a decade ago, relatively few 
states have evaluated the effectiveness of the training activities in improving adult participants' 
likelihood of employment, and in increasing their earnings if they are employed. State-level 
evaluations are particularly important since states have major responsibilities for managing the 
JTPA programs and for establishing policies which affect the operations and performance of 
Service Delivery Areas. 

A few states have undertaken non-experimental evaluations of their programs: a compariso 
of the employment and earnings of JTPA participants after they left the program with the 
employment and earnings of a comparison group who did not enroll in JTPA. Data on earnings 
are obtained from Unemployment Insurance (UI) wage records. For the most part however, 
states have relied on descriptive data on post-program outcomes to assess the effectiveness of 
their programs. This approacn can produce misleading results, as this report indicates. 

To encourage state-level evaluations of JTPA, the Corp.rnission and the State of Utah 
co-sponsored this report on the effectiveness of JTPA Title 11 programs for adults in Utah. The 
technique used is non-experimental and the comparison group was selected ^rom registrants 
with Utah's Employment Security. In a step-by-step approach, the report juments how 
non-experimental techniques can be applied and several specific technical issues can be 
addressed. 

The report has produced two major findings. First, states need not rely on descriptive 
information to assess JTPA's effectiveness. It is feasible to use UI wage-record data and 
non-experimental techniques to evaluate JTPA programs for economically disadvantaged 
adults. Smaller states may wish to pool their data to achieve greaf^^r statistical precision and to 
conserve on resources. Also, these evaluations require that state JTPA offices work closely with 
state UI offices to ensure that the necessary data are maintained. Long-term evaluations of 
program impacts need UI wage record data spanning many years, from 2 or more years prior 
to participants' enrollment in JTPA training to 2 or more years following enrollment. 

The second major finding from this report is that non-experimental techniques produce 
results on post-program outcomes that are highly relevant to policy makers. This particular 
finding comes at a time when the majoralternativesourceof information on JTPA's effectiveness 
is based on a national experiment, funded by the U.S. Department of Labor. The recently 
released results from the "National JTPA Study" have become the benchmark for assessments 
of JTPA's effectiveness. 

However, due to difficulties encountered when the experiment was being implemented, results 
from the National JTPA Study ought not to be considered definitive evidence of the impact of 
receipt of JTPA training. Whi'.e the National Study was designed to be a scientifically rigorous 
social experiment, complete with random assignment of individuals into either a "treatment" or 
a "control" group," it did not estimate the "net impact" of receipt of JTPA training. That is, the 
study did not fully answer the question: Did adults who were in JTI^A training programs fare 
better than those who did not receive training? 

Instead, the treatment group consisted of persons assigned to a JTPA training activity; some of 
them subsequently enrolled in another JTPA activity rather than the one to v;hich they were 



assigned, while others did not enroll in any activity. The control group consisted of persons 
assigned to a JTPA activity, but not allowed to enroll in that activity; some subsequently i eceived 
JTPA-funded job search assistance while others received training through a non-JTPA funded 
source, 

in contrast, the results from this study of Utah, although they arc from only one state, shed light 
on the effectiveness of receipt of JTPA training. For men, participation in Utah's JTPA 
occupational classroom training and on-the-job training programs was associated with an 
improved likelihood of employment more than 1 year after participants left the training. For 
women, participation in on-the-job training was associated with a greater likelihood of being 
employed 1 year later. For those who were employed, the strongest earnings payoffs were 
associated with on-the-job training (for women) and occupational classroom training (for men). 
The results are broadly consistent with those of the National JTPA Study. In particular, 
on-the-job training appears to be a key training activity. 

In addition, participants in Utah who had completed their training and had been placed by the 
program, experienced earnings gains 2 years later. This last finding highlights the potential 
importance of JTPA's job placement activities. 

The Commission believes that this report provides strong encouragement to states desiring to 
evaluate their JTPA programs. It also provides important information on the effectiveness of 
JTPA programs for improving the labor market position of economically disadvantaged adults. 



JOHN C GARTLAND 
CHAIRMAN 
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■ CHAPTER ONE: Introduction 



Objective Of Study 

Since the 1960s, the federal government has sponsored employme it and training programs to 
assist economically disadvantaged youth and adults. Their goal is to improve participants' 
prospects in the labor market - to increase their likelihood of employment and if employed, to 
increase their earnings. During this period, over 50 federal training programs have spent 
roughly $100 billion under five fegislative Acts -- the Area Redevelopment Act of 1961 (ARA), 
the Manpower Development and Training Act of 1962 (MDTA), the Economic Opportunity Act 
of 1964 (EOA), the Comprehensive Employment and Training Act of 1973 (CETA), and the Job 
Training Partnership Act of 1982 (JTTA). 

Unlike its predecessors, JTPA has had an evaluation component tied explicitly to its legislative 
foundation. The 1982 Act state? that federally sponsored employment and training efforts are 
"investments in human capital"; in Section 106(a) it further states: 

... it is essential that criteria for measuring the return on the investment be developed; 
and the basic return ... is to be measured by the increased employment and earnings 
of participants and the reduction in welfare dependency. 

For the first decade under JTPA. states have cnly been able to measure the effectiveness of Title 
Il-A programs for the economically disadvantaged against a set of national stanc'ards, which 
may be adjusted to reflect varying client characteristics and local economic conditions in the 650 
f)lus local Service Delivery Areas (SDAs) across the nation. These measures provide only 
relative outcome information across the SDAs - not measures of return on investments, 
however. The latter can only be determined by comparing the outcomes of participants against 
what would have occurred had enrollees not participated in JTPA - the "net impact" of training. 

The major objective of this study is to provide states with a methodology for evaluation that 
meets the guidelines of the Act. The methodology proposed is carried out using as a prototype 
the state of Utah's JTPA II-A adult program for participants entering during PY 1987. This 
approach is non-experimental in nature; it permits evaluation of on-going programs without 
disrupting nom^al operational behavior. Unlike social experiments where eligible clients arc 
randomly assigi^ed to a treatment group or to a control group that is denied services, no one 
eligible for JTPA assistance need be turned away when using this approach. It is an evaluation 
methodology that utilizes currently available administrative data bases - JTPA data file 
supplemented with state Employment Seairity registrant files and Unemployment insurance 
(UI) quarterly earnings records. As such, the methodology shortens the evaluation cycle and is 
economical. 

Finally, it is a methodology that captures the behavior of clients and program operators within 
its stalislical design so that administrators cannot only learn if a program "works," but how the 
program works, and for whom the program works. The information available to program 
administrators in this non-experimental evaluation is encompassing. The more years of 
individuals' post-training UI earnings records that are archived, the longer tho experience can 
be measured. The approach can be used for all JTPA programs as well as other state-supported 



programs such as higher educaHon, welfare, or even for evaluating the employment and 
earnings effects of incarceration in a state's penal system. 



As discussed in detail throughout the study, this non-experimental approach is only now 
coming into its own. Previous work utilizing tnis methodology was plagued by a fundamental 
problem, termed selection bias, which seriously diminished the merit of eariier evaluations 
using the non-experimental approach. In particular, the non-experimental approach compares 
the outcomes of those receiving assistance Uhe treatment group) with similar individuals who 
did not receive the assistance (the comparison group, analogous to an experimental control 
group). 

Two problems, heretofore, have plagued such studies: the comparison group identified was 
often dissimilar from the treatment group, and the process by which individuals sorted 
themselves into either the treatment group or comparison group was not a random process, as 
IS characteristic of social experiments. If, for example, individuals who self-select into JTPA are 
less motivated, have fewer job contacts, or have greater not-easily-identified employment 
barriers than those in the comparison group, the evaluation will yield "downward biased" 
estimates of the net impact of JTPA. In this case, JTPA enrollees may experience lower earnings 
than the comparison group - not because the program made them worse off - but because the 
mdividuals who selected into JTPA arc "different" from those who chose not to enroll. 

Over the last 10 years, improvements in statistical modeling have reduced the suspected 
selection bias of the non-experimental approach. New methods have been developed to capture 
better unobserved factors thai give rise to selection bias. Also, new tests have been developed 
to yield better comparison groups which are less susceptible to selection bias. These 
improvements now make it possible for states to carry out their own evaluation of employment 
and training programs initially envisioned in the JTPA /.ct. 

A secondary objective of this study is to raise the status of this non-experimental approach 
relative to social experiments that have been in vogue since the 1970s. A major experimental 
evaluation of JTPA - the National JTPA Experiment (NJE) - has been undertaken since 1986, 
and Its 18-month interim report is just now becoming available to the public.^ The common 
perception of this experiment is that its results must reflect the "true" net impact of JTPA since, 
after all, the social experiment randomly assigned individuals to either the treatment or control 
>roup to avoid selection bias. 

The results of NJE are not very flattering of JTPA. Adult net impacts are perceived to be 
"marginal," while those of youth are "disturbing," especially for young males who supposedly 
are made worse off by participating in JTPA. ^ 

The problem is that the JTPA experiment was not able to be carried out as originally planned 
resulting in selection bias problems of its own. (Appendix A details these problems.) The 
seriousness of these selection biases is unknown and may, in fact, never be able to be quantified. 
Such a state of affairs suggests that major policy changes should not be based upon the NJE 
results, which may be seriously misleading, or at best uncertain. In truth, after a decade of JTPA 
wo still know very little regarding the effectiveness of our country's major employment and 
training program, as called for in the original legislation. 

Tliis report envisions that state JTPA administrators will begin serious evaluations of their own 
programs. At best, social experiments can indicate if JTPA works. Careful non-experimental 
evaluations of state JTPA programs can provide ser\nce providers with information to improve 
program effectiveness: the why, how, and for whom JTPA "works." This, after all, was what 
Congress envisioned in Section 106 of the 1982 JTPA legislation. 
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study Outline 



Chapter Two below, sets the stage for the non-experimental empirical analysis. It discusses the 
reasons why this study was undertaken, including the issues raised in the evaluations of 
employment and training programs undertaken in the early 1980s that led to the Department 
of Labor's NJE. It also introduces the "non-experimental" evaluation technique used in this 
study, and describes how better comparison groups can be identified and new statistical 
techniques can be applied to derive "quasi-net impact" estimates free of selection bias. 

Chapters Three and Four delve into issues involved in selecting a comparison group. Chapter 
Three describes key demographic and labor market characteristics of JTPA Title II adult 
participants in Utah, who are the focus of the analysis - and whose characteristics are used to 
match similar individuals from the Employment Security file. Chapter Four discusses the 
process of selecting this comparison group. By its very nature, this procoss is technical and the 
discussion is lengthy. This chapter is intended primarily for evaluators who would be 
implementing a study similar to the one in this report. 

Chapter Five presents the results of the Utah non-experimental evaluation. First, simple 
descriptive statistics on outcomes are given and are compared with outcomes derived from a 
rigorous statistical analysis. This discussion indicates the extent to which simple descriptive 
data can be misleading. Two topics are highlighted in the results of the statistical analysis: the 
probablistic nature of estimates of the post-program employment and earnings impacts, and the 
usefulness of the analysis for identifying "hard to serve" participants. 

Chapter Five then turns to a comparison of alternative estimates of program impacts. The most 
loliable ones - based on the best comparison group, and using the most extensive data and the 
most sophisticated statistical techniques - are systematically compared with others. The other 
results use, respectively, another comparison group, less extensive data, and less rigorous 
techniques. These comparisons demonstrate that results can differ substantially depending 
upon the approach take:i. While it is necessary to be careful in using a non-experimental 
approach in program evaluation, the technique can be usefully applied with carefully chosen 
comparison groups and properly specified program impact models. 

The concluding Chapter Six highlights findings and discusses their implications for policies to 
in-iprove both the operation of employment and training programs and states' evaluation of 
them. 



Endnotes 



1. Blcx)m,ctal.(1992). (An Executive Summary, pre-rclcascd in May of 1992, has been available 
to the public. Copies of the Interim Report, Blcom, et al. (1993), began to become available in 
January 1993. 

2. The popular press has interpreted the findings of the 1992 ExecuHve Summar>' in quite 
misleading ways - further adding fuel to the fire regarding faulty percepHons of JTPA program 
effecti\'enoss. See for example, James Bovard, "Clinton's Summer Jobs Sham," The Wall Street 
lournal, March 5, 1993; and Spencer Rich, "Runaway Training: Why Educating Poor, Unskilled 
Workers Won't Land Them Jobs," Washington Post. February 7, 1993. 



■ CHAPTER TWO: Prior Evaluations 
Of Federal Training Programs For 
Economically Disadvantaged Adults 

As indicated above, JTPA was the first federal employment and training program to be viewed 
as an investment in human capital and one that was to be evaluated according to a rate of return 
measure of performance. This meant program performance was to be evaluated on economic 
efficiency grounds, in addition to its effectiveness in Improving opportunities of the 
economically disadvantaged in the labor market. 

Section 106(a) of the Job Training Partnership Act (JTPA) of 1982 called for evaluation analysis 
of program performance so Congress and employment and training officials could better 
ascertain the effectiveness of JTPA in comparison with other government employment services 
and training alternatives. The U.S. Department of Labor (USDOL) was instructed to analyze 
JTPA program performance according to: (1) increased participant employment and earnings, 
(2) decreased income support costs (i.e., welfare and unemployment), and (3) increased tax 
revenues. 

The purpose of this chapter is to trace the historical background of evaluations of employment 
and training programs for economically disadvantaged adults, so that the methodological and 
statistical grounds of the approach used in this study will be understood in the broader context 
of evaluation research. First, the chapter distinguishes between program-effectiveness 
measures related to performance standards and those measures related to net impacts, which 
are at the heart of program evaluation. Next, the non-experimental evaluations of JTPA's 
predecessor programs are discussed from the perspective of their weaknesses, which led to their 
abandonment by USDOL for the more theoretically appealing experimental approach. With 
this as background, the current NaHonal JTPA Experiment (NJE) is described, along with 
suspected selection biases which resulted from implementation problems the experiment 
encountered. 

Finally, advances in the non-experimental approach are briefly outlined; they provide the 
rationale for resurrecting non-experimental approaches to evaluating employment and training 
programs. These advances should be especially appealing to states that either cannot afford the 
more costly experimental evaluations and /or choose not to subject on-going programs to the 
disruptions caused by an experimental approach. 



Program Outcomes Versus Program Net Impacts 

Unlike the observed program outcomes, which can be directly compared to JTPA (adjusted) 
performance standards, program net impacts must be estimated using statistical modeling 
techniques. For example, the program's net impact on post-training earnings is calculated as 
the differential between actual post-program earnings of participants and that estimated with 
statistical models of projected earnings had participants not enrolled in JTPA. (The latter cannot 



be observed, rather it is "counterfactual" based upon what enrollees would have experienced in 
the absence of training.) 



Both measures give different information as to program effectiveness. Program outcomes 
measured against performance standards can be used, for example, to describe the distribution 
of Service Delivery Area (SD A) performance across a state. In contrast, program net impacts can 
be used to determine if the state JTPA system as a whole improves the economic welfare of 
enrollees. Cleariy both are of importance to employment and training administrators. The 
objective of this study is to examine the feasibility of deriving state-based net impacts of program 
performance. 

The difference in the two measures of program perforrrunce has recently come to light with the 
18-month interim results of NJE, first made public in May of 1992. In the same year the random 
assignment process of the NJE was being implemented (Program Year 1987), information on 
outcomes of JTPA's adult programs appeared to be quite favorable. For example, the actual 
Adult Entered Employment Rate (the percent of adult terminees who are in unsubsidized jobs) 
for the system as a whole was 72%, compared to the national standard of 62%. Also, the actual 
Average Wage at Placement (the average wage at termination of those employed) was $5.11, 
compared to the national standard of $4.91.^ 

Following 10 years of states' using such in formation as this as a guide to program effectiveness, 
the NJE gave JTPA its first "score card" on how the system appeared to be operating. As noted 
in Chapter One, the news was quite sobering. Adult women assigned to JTPA were estimated 
to increase their chances of being employed by 2% and to increase annual earnings by $1 a day 
over the control group not assigned to JTPA. Adult men, while mor.^ likely to be employed, did 
not experience significant earnings gains during the 18-month period.^ Clearly the two 
measures of program effectiveness tell us very different things. 

The fact that evaluations of relative SDA performance are just now being supplemented with 
net impact evaluations is not a result of neglect or oversight. Rather it is the result of a 
long-standing and heated debate within the technical evaluation community of how best to 
derive net impact estimates of federal training programs. 



Pre-JTPA Non-Experimental Evaluation Studies 

Originally, USDOL administrators envisioned that it would be best to sponsor 
non-experimental evaluaHons of the new JTPA program. Social experiments, which had come 
into vogue during the 1970s, were used mainly to evaluate expected outcomes of demonstrations 
which used yet-to-be implemented policies.^ At that time was thought best to evaluate JTPA 
as it normally operated rather than under experimental o iitions of never-tried policies. 

US1X)L thought that consistent and reliable estimates of JTPA program impacts could be 
derived through non-experimental studies that used improved enrollee and comparison group 
surveys. The Continuous Longitudinal Manpower Survey (CLMS) was to be replaced with the 
Job Training Longitudinal Survey (JTLS) and the Current Population Surveys (CPS) was to be 
supplemented with the Survey of History of Work (SHOW), SHOW would provide researchers 
with a national survey of non-participants, which could be used to develop improved 
comparison groups for analysis of program impacts. 

This nou-experimenta! approach had its roots in the late 1970s when USCOL had cc:ntracted 
with Westat, Inc. to develop the CLMS for use in analyzing CETA. In these non-experimental 
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studies, program participants were combined with a sample of low income individuals selected 
from the March survey of the Current Population Surveys (CPS). The latter was chosen so that 
"similar" comparison groups could be derived for use in the net impact evaluation studies. Early 
analysis of CLMS by Westat using cohorts enrolled in the mid-1970s resulted in a range of 
estimated program impacts too wide to be considered reliable, however. 

From 1980 - 1982, the USDOL decided to fund outside studies of CETA using alternative 
modeling specifications and different choices of comparison groups for adult and youth groups 
separately. The intent of these and related studies, was to determine if non-experimental 
research could be elevated to yield stable and reliable estimates of program impacts for use by 
USDOL. 

The results of six program impact studies for economically disadvantaged adults are highlighted 
in Table 2.1."^ These studies evaluated the earnings experience of CETA participants who 
enrolled between 1975 and 1976 and are based upon Social Security Administration (SSA) 
earnings.^ They sampled the national population of CETA participants and chose various 
comparison groups from the national CPS data. 



TABLE 2.1 



PROGRAM IMPACTS FOR SELECTED STUDIES OF CETA ADULTS 



STUDY 


BLOOM AND 
MCLAUGHLIN 


BASSI* 


DICKINSON, VVEST, 
AND JOHNSON 


GERAO 


CHARACTERISTIC 


(1982) 


(1984) 


(1983) 


(1984) 


(1986) 


(1984) 


STUDY DESIGN: 














TRAINING DATES 


1975-1976 


1975-1976 


1975-1976 


1975 


1976 


1975-1976 


POST-ENROLLMENT DATES 


1976-1977 


1978 


1979 


1977 


1978 


1979 


EARNINCS DATA SOURCE 


SSA 


SSA 


SSA 


CETA 
SURVEY 


SSA 


SSA 


SAMPLE SIZE: 














Treatment Group 




8217 


8217 






3896 


Comparison Group 




4139 


4139 


5890 


11582 


3384 


PROGRAM IMPACTS: 














TOTAL PROGRAM: 














VVomen 


800 TO 1300 


1621/988 


987/1096 


905 


-26* 


NA 


Men 


200 


509/132 


414/343* 


-817 


-1123 


NA 


CLASSROOM TRAINING: 














Women 


800TO1400 


NA 


910/1030 


1320 


NA 


1201 


Men 


300» 


NA 


346''7292* 


-470* 


NA 


372* 


ON-THE-JOB TRAINING: 














Women 


700 TO 1100 


NA 


214/546* 


1107 


NA 


882 


Men 


300* 


NA 


762/887 


-282* 


NA 


-588 



• Impacts shov/n for white/minority ethnic groups .espectively. 
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Since the studies evaluated programs over 15 years ago, including the highly inflationary period 
of the late 1970s, all ifindings should be adjusted for inflation before placing boundaries on the 
range of estimated program impacts. These adjusted figures are shown below, stated in 1987 
dollars/ 



CETA PROGRAM IMPACT RANGES STATED IN 1987 DOLLARS'^ 





WOMEN 


MEN 


TOTAL PROGRAM 


+$1,000 TO +$2,200 


-$2,000 TO +$900 


CLASSROOM TRAINING 


+$; ,400 TO +$2,600 


Not Significant 


ON-THE-JOB TRAINING 


+$300 TO +$2,000 


-$1,000 TO +$1,200 



* Figures dorived from program impacts stated in current year prices of Table 2.1. 



As noted, a wide range of estimated program and activity impacts was found. Two major 
findings are important, however. First, adult women appear to ha ve benefitted more than men 
on average, as well as in the more intensive classioom and on-the-job training activities. For 
example, the estimated program impact for women varied between +$1,000 and +$2,200 per 
year; that for men varied between -$2,000 and -; $900 per year.^ Secondly, adult women 
experienced greater earnings impacts through classroom training co'npared with on-the-job 
training programs; the opposite was found for adult men. In short, it became clear from these 
early evaluations that sole reliance on enrollee-comparison group research methodologies could 
produce a wide range of program impact estimates. 

Accordingly, after ]TPA was enacted in 1982, a panel of experts was convened by USDOL to 
review plans for examining JTPA program impacts and to recommend ways to deal with the 
unresolved methodological issue of selection bias. The panel made their official 
recommendations to USDOL, and in 1985 under strong pressure from some of the most 
respected labor economic researchers, the Department abandoned the JTLS/SHOW surveysand 
rc-allocated their funds to classical field experiments.^ Based upon the panel's 
recommendations, it issued Request For Proposals (RFPs) for a research design, accommodating 
both non-experimental and experimental approaches, to derive estimates of JTPA program 
impacts. In a departure from past methodology, the National JTPA Experiment (NJE) would 
center on the random assignment of JTPA Title II-A adults and out-of school youth applicants 
to cither a treatment or a control group. 

As part of the NJE, efforts were also made to construct an improved comparison group from an 
eligible non-participant sample so estimates of the selection bias could be derived. The logic for 
developing both experimental and non-experimental evaluation designs was based on both 
research methodology and cost considerations. It was believed that the best estimates of 
program impacts could be found by undertaking an expensive classical experiment involving 
randomly assigned treatment and control groups. The results from this study could then be 
used as a bench mark against which to compare various non-experimental impact estimates 
derived from the same treatment group and a specially designed eligible nc n-participant (ENP) 
comparison group. As such, the experimental control group could be used as a standard for 
choosing the best ENP comparison group from the non-participant sample. Knowledge of how 
to choose an appropriate comparison group, along with an understanding of the sensitivity of 
various comparison groups to selection bias, could guide future non-experimental studies. 
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They would indicate the degree and direction of bias remaining, if any, in the non-experimental 
modeling designs. 

From a practical point of view, USDOL understood that the vast majority of future JTPA 
program impact analyses, if ever undertaken by states, would of necessity follow the less costly 
non-experimental designs. The program impact estimates of experimental design would act as 
beacon in the dark at which all other non-experimental studies could aim. 

Over the years, many program evaluation researchers have remained rather skeptical of 
non-experimental studies, because of their likelihood of generating net impact estimates that 
reflected severe selection bias. Prior to this study of Utah's economically disadvantaged adult 
program, eleven states had analyzed Title II-A adult programs, but only two states used 
non-experimental net impact techniques.^^ Both of these studies chose random samples of state 
Employment Service applicants for comparison groups and utilized relatively simplistic 
analytical models to address the selection bias issue. 

Indiana evaluated 922 trainees enrolled during Program Year 1983 against 1 M6 individuals 
from the Employment Service (ES) comparison group stratified by race-genuer target groups. 
For the second year following entry, annual earnings impacts of $1,000 for white women and 
$1,200 for AFDC recipients were found; program impacts for males were repor ' ^d to be $1,400. 

Similar findings were reported by Nevada for their Title II-A programs during Program Year 
1985. That stage's study evaluated the earnings performance of 817 trainees against 5,300 
individuals randomly chosen from the ES applicant file.^^ During calendar year 1987, overall 
program impacts from $1,400 to $1,700 were reported for adult men in three alternative 
analytical models, and from $630 to $930 for adult women. 

Contrary to prior CETA studies, adult men experienced greater program impacts than women 
in both state-based studies. Since these findings have not been subjected to more rigorous 
selection bias tests as used in this study, the net impact estimates should be interpreted with 
caution. 



Experimental Evaluation Studies 

While historically the majority of evaluations of employment and training programs have used 
non-experimental methodologies, a number of social .experiments have also been undertaken 
to evaluate a host of federally funded education, welfare, and training programs. One study 
estimates that neariy $1 billion has been speni over 100 such experiments in the past 25 

12 

years. 

For the purposes of this study, however, only the National JTPA Experiment (NJE) - which 
evaluated the post-training employment and earnings of 16 selected SDAs during the late 1980s 
- will be described. This multi-million dollar experiment was patterned after eariier classical 
random assignment models of social program evaluation. 

A hypothetical, and ideal, design structure of such an experiment, with random assig-nment at 
multiple stages, is given below in Figure 2.1. In this model, there are three stages of random 
assignment; in combination, they would yield unbiased JTPA program impacts in total and also 
by major activities for the national JTPA system. 



FIGURE 2.1 

HYPOTHETICAL MULTI-STAGED JTPA 
RANDOM ASSIGNMENT EXPERIMENT 



JTPA 
UNIVERSE 








SDAs IN 
EXPERIMENT 







/ R/A 







TREATMENT 
GROUP 








SDAs OUT OF 
EXPERIMENT 



CONTROL 
GROUP 



















CT 




OJT 




OA 



R/A - Random Assignment 
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The first random assignment occurs across the universe of SDAs; it is designed to ensure the 
results arc not biased by non-random selection of SDAs that participate in the experiment. The 
second random assignment occurs across eligible participants; it is designed to ensure that the 
results are not biased by the choices of the eligible participants to enroll or not to enroll in the 
program (or to be selected, or not selected, by the JTPA staff) . The final random assignment is 
across enrollees and into major activities; it is designed to ensure the results are not biased by 
the choice of enrollees to select (or be placed) into one of the three major categories of JTPA 
activities, in this figure, there are three major activities: occupational classroom training (OCT), 
on-the-job training (OJT), or other activities (OA).^^ 

Non-random selection - or "choice" - is not permitted in this hypothetical construct so that 
"true" program impacts, i.e. free of selection bias, can be estimated. According to the figure, the 
total JTPA program impact is measured as the difference between the mean post-program 
outcomes of those assigned to the treatment (T) group less the mean outcomes of those in the 
control (C) group (T minus C). Similarly, JTPA program activity impacts are measured as the 
difference between the mean outcomes of those assigned to each activity group less the mean 
outcomes of control group members (OCT minus C; OJT minus C; and OA minus C). 

In reality, implementation of the NJE did not follow such a laboratory-like design. (A detailed 
description of the NJE and its underlying design problems are given in Appendix A.) 
Breakdowns occurred at all three stages of randomization. First, so few of the SDAs identified 
on a random basis agreed to participate in the experiment that the first stage random assignment 
was eliminated during implementation. Instead, USEXDL agreed to offer increasingly higher 
"inducement fees" for participation and still was only able to contract with 1 6 out of the originally 
planned 20 SDA silcs.^^ From the beginning, it became evident that the experimental results 
should not - and could not - be interpreted as reflecting "the" JTPA system. 

Second, all eligible participants were given some "minimal" amount of JTPA assistance - in the 
form of testing, assessment, and varying degrees of job search and referral assistance. In essence, 
the control group did receive at least some level of JTPA treatment. Also, some members of the 
coiUrol group received training and other assistance from non-JTPA sources. Thus, the program 
impacts were downward biased. That is, the impact of those included in the treatment group 
was compared to a control group that was not denied services provided by JTPA and other 
sources. The extent of this bias varied by individuals and types of services made available 
through Title II-A and other sources. 

Third, a small - but significant - proportion of the treatment group participants enrolled in 
activities different from those to which they were initially assigned. Either the individuals 
self-selected out of their initial assignment, or they were selected out of the initially assigned 
activity due to changes in program funding /availability or for other reasons. In either case, 
non-random selection partially characterized activity enrollment, which resulted in biased 
estimates of activity enrollment impacts. 



Reconcaiation Of Findings Based On Alternative Methodologies 

The justification for using the non-experimental evaluation approach is based upon a 
reconciliation of findings using experimental and non-experimental methodologies, hi the 
mid-1980s, evaluation experts tried to duplicate estimates of experimental net impacts using 
non-experimental statistical techniques. They were seeking to determine if results would be 
seriously biased when non-experimental models were used to simulate experimental net 
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impacts, if such a result were found, this would demonstrate that program net impacts should 
only be derived through randomized social experimentation. 



Selection Bias of Non-E xperimental Studies . To test the usefulness of non-experimental 
models, two independent researchers tried to simulate the findings of an eariier demonstraHon 
experiment which served adult women who were long-term AFDC recipients. The experiment, 
the National Supported Work (NSW) Demonstration, had been ran in the mid- to late 1970s and 
had previously been evaluated under a USDOL research conh-act. (More detailed discussions 
of these studies are given in Appendix B.) 

These two follow-on studies found that non-experimental models, when applied to the 
experimental NSW design, produced a wide range of quasi-net impact estimates. The results 
supported the view that non-experimental results are seriously biased.^^ 

In the first study, LaLonde derived two sets of comparison groups from the Panel Study on 
Income Dynamics (PSID) and the Current Population Survey (CPS), which were used as 
surrogates for the experimental control group. Social Security Administration (SSA) earnings 
data were appended to both treatment and comparison groups for measures of post-program 
outcomes. Results using the PSID comparison group yielded quasi-net earnings impacts from 
two to three times as high as the experimental control group; those using the CPS varied from 
smaller estimated quasi-net earnings impacts to impacts over twice as high as the experimental 
results. 



In the second study, Fraker and Maynard derived six comparison groups from another CPS file 
and also appended SSA earnings data to the treatment and comparison groups. Once again, 
the non-experimental approach yielded a wide range of quasi-net impact estimates, from "not 
significant" to estimates twice as high as that from the experimental net impacts. 

Bias-Free Non-Experim ental Models . These findings temporarily sealed the fate of 
non-experimental evaluation methodologies. However, the-e were two important weaknesses 
in the studies that affected their results. 

First, the comparison groups selected differed greatly from the treatment group in the NSW 
demonstrations. In the ULonde study, only 30% of the PSID comparison group were AFDC 
recipients; as were 2% of the CPS sample. The pre-intervention earnings of these comparison 
groups were sometimes 10 times as high as the treatment group. 

More importantly, the non-experimental models were later shown to be misspecified, which 
produced a selection bias. In a key study in evaluation literature, Heckman and Hotz later 
developed a series of tests on pre-training and post-training earnings to identify selection bias.^^ 
By comparing the earnings processes of the non-experimental comparison groups with the 
experimental control group in the NSW demonstration, the authors were able to identify which 
non-experimental models were subject to selection bias. With the misspecified models rcmoved 
from analysis, the authors then estimated the quasi-net impacts using only those comparison 
groups that passed the tests. The use of more complex statistical models, which could capture 
the impact of unobserved factors on program selection, along with improved comparison 
groups resulted in quasi-net impact estimates similar to those of experimental NSW study. 

For the first time, use of non-experimental evaluation techniques had simulated results of a 
randomized social experiment. The authors were careful to point out, however, that before 
non-experimental quasi-net impact estimates can be free of selection bias, much care must be 
taken in deriving similar comparison groups and in utilizing appropriate selection-bias tests.^^ 
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Ideally, states would need to archive three or more years of individuals' Unennployment 
insurance (UI) earnings data ~ both prior to and following intervention — in order to use the 
nnore stringent selection bias tests for non-experinnental evaluation nnodels. Nevertheless, a 
non-experinnental approach to program evaluation can produce results that are technically 
sound, and thus useful to policy makers. It is this approach that is taken in the remainder of this 
report. 
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Endnotes 



1 . U.S. Department of Labor, Division of Performance Management and Evaluation, Office of 
Strategic Planning and Policy Development, "Summary of JTQS Data for JTPA Title II A and III 
Enrollments and Terminations During PY 1987 (July 1987-June 1988), Washington, D.C., 
December 1988, p. 16. 

2. These figures are derived from Bloom, et al. (1993). 

3. Perhaps the most widely known experiments of this type were those that estimated the 
expected outcomes of income maintenance progranns for the poor. These experiments were set 
up in four areas throughout the country to determine the labor force participation responses of 
men and women who were given varying amounts of income maintenance. These "negative 
income tax" evaluations were analyzed in: Kershaw and Fair (1976); Hannon, Tuma and 
Groevenveld (1977); and later in Cain and Wissoker (1990). 

4. Two studies were completed by Westat, Inc., but did not estimate program im.pacts 
separately for adults, as is the focus of this study; see: Westat (1981 ) and Westat (1984). As such, 
the results of the studies are not included in the table. A survey of the major CETA evaluation 
statistical methodologies and their findings are summarized in Bamow (1987). 

5. At this time, longitudinal earnings files for individuals had to be constructed from 
individual Social Security Administration (SSA) records, which resulted in lengthy delays. For 
example, the USEXDL-sponsored evaluations of CETA, completed in 1984, evaluated CETA 
programs which operated between 1975 and 1976. 

6. A major problem faced by these early evaluations was choosing comparison groups which 
were similar to the CETA participants in both observed and "non-observed" factors. In addition, 
the selection of the comparison groups based upon pre-^^nrollment earnings, when successful, 
resulted in sampling contamination as CETA participants were included in the CPS-based 
comparison groups. One study [Dickinson, West, and Johnson (1984)] estimated that up to 
one-fpurth of the comparison group was composed of CETA treatment groups. 

7. The year 1987 is chosen because this is the program year analyzed in the Utah Title II-A 
adult program in this study. 

8. The negative findings for adult men may be attributed to the program diverting participants 
from productive job search or employer discrimination against hiring and /or paying (ZETA 
participants. In spite of lower post-training earnings observed for the average participant, 
individual participants may still enroll in such program.s due to their uncertainty of finding jobs 
outside of CETA as opposed to their certainty of receiving training and/or stipends in CETA. 

9. See: Stromdorfer,etal. (1985). 

10. See Reed (1986) and Hanna and Turney (1988). A Title III Dislocated Worker program net 
impact evaluation by Bowman (1988) was also completed for Utah. A national survey of state 
evaluation projects for Title 11-A is found in Figueroa and Misch (1990). 

1 1 . See: Hanna and Turney (1988). 

12. Greenbcrg and Shroder (1991). 
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13. The use of random assignment models to evaluate social programs was an outgrowth of 
early agricultural experiments proposed by Fisher (1935). Needless to say, the extension of such 
experiments to evaluate social programs, characterized by human behavior, has resulted in 
strong proponents as well as opponents to such applications. The debate in this heated 
controversy is covered, among other places, in: Campbell (1970), Rivlin (1971), Cook and 
Campbell (1979), Lucas (1981), Burtless and Haveman (1984\ Hausman and Wise (1985), 
Burtless and Orr (1986). Greenberg and Robins (1986), Sawhill (1988), Hcckman (1992), and 
Levitan (1992). 

14. A fourth stage of random assignment could be added to this hypothetical model; it would 
be at the final stage, for JTPA placement services. It is excluded here for ease of exposition. 

15. These categories are the same as those specified in the NJE, which allowed for multiple 
activity sequences, where individuals were classified according to their major activity if more 
than one was received. 

16. It may be noted that the NJE design altered this hypothetical design by instituting 
randomization between treatment and control groups after individuals were assessed, tested, 
and assigned to an activity. More importantly, the assignment to an activity was not done on a 
random basis, since this procedure would have assigned individuals to an activity which might 
not have been appropriate or optimal. As such, the NJE design precluded unbiased estimates 
of inter-activity impact comparisons (i.e., OCT-OJT; (X:T-0A; or OJT-OA), since the assignment 
across activity groups was based up)on a non-random process. 

17. Initially SDAs were offered $40,000 to participate in the experiment. This figure was raised 
to an average of $170,000, with one site being paid $300,000 to participate. These figures are 
taken from Hotz (1992), p.95. 

18. As a matter of feasibility, the NJE was constrained to only a single level of random 
assignment -- which was placed after systematic assignment to one of the three major activity 
sequences. Had the non-random selection across activities not occurred, the experiment could 
have produced unbiased estimates of activity enrollment impacts. Since this was not possible, 
unbiased estimates (ignoring the previous two sources of bias) of only the assignment to an 
activity - as distinct from actual enrollment in an activity - was possible in the experiment. 

19. See LaLonde (1986) and Fraker and Maynard (1987). 

20. See Heckman and Hotz (1989). 

21. A recent study by Friedlander and Robins (1992) used only the most simplistic selection 
bias tests involving comparison groups that were computed from various experimental control 
groups of mostly single female AFDCZ recipients. The binary outcomes of being employee . md 
receiving welfare during the post-training period were analyzed. These tests resulted in 
uncertain outcomes, which were limited by the nature of the simplified selection processes 
tested and the discrete nature of the program outcome measures. 
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■ CHAPTER THREE: Utah JTPA Title 
II-A Adult Participant Characteristics 



This chapter begins the non-experimental approach to evaluatioiv by describing the 
demographic characteristics and prior work experiences of the economically disadvantaged 
adults enrolled in Utah's JTPA Title II~A program during Program Year (PY) 1987 (1,864 
individuals). The study includes all applicants aged 22 to 54 who had never previously enrolled 
in one of three major activities during that program year (classrooia training in occupational 
skills; on-the-job training; and other activities, composed mainly of job search assistance). This 
number includes participants who completed and were placed into unsubsidized jobs in 
addition to those who completed but were not placed into jobs, and tl ose who dropped out. 

The description of the participants and their prior work experiences ' s important because the 
observed participant characteristics will later be used to identify oimilar comparison groups 
from the files of Utah Employment Security (ES) participants. These characteristics are used to 
account for observed differences between JTPA participants and non-participants in the net 
impact statistical models of the study. 

In the first section, a general description of Utah's JTPA programs is given to emphasize the 
important role the Title II-A program for economically disadvantaged adults plays in the state. 
In the next section, the PY 1987 Title II-A adult participants are described. In the last section, 
the pre-entry work history of Utah's Title II-A adult participants is described in more detail 
because this work history is fundamental to th J identification of experimental comparison 
groups used to derive program impact estimates in this study. 



Utah's JTPA Programs During PY 1987 



The PY 1987 adult participants selected for analysis were a group whose employment and 
earnings history can be tracked with Utah's Unemployment Insurance (UI) wage records over 
a 4-year pre- to post-entry period. That is, historical UI wage files in Utah, available when this 
study began, include all quarters continuously from the third quarter of 1984 through the second 
quarter of 1990. The files allow for 8 full quarters of pre- and post-entry work experience to be 
identified for all participants prior to entry into JTP A.^ 

During this period, over 5,000 individuals entered all Utah JTPA programs for the first time. 
Nearly one-half (46%) were eligible for the Title II-A adult program (which is composed of 
individuals aged 22 and older who were certified as economically disadvantaged). Title II-A 
adults meeting the federal/state guidelines are assessed and placed mto one of three 
employment and training programs — occupational classroom training (OCT), on-the-job 
training in the private sector (OJT), and other activities (OA), which as noted earlier, in Utah 
consists predominantly of job search assistance (JSA) for those vith readily marketable skills. 
As shown in Figure 3.1, Title II-A adult programs constitute by far the largest federal-state 
employment and training activity; the second largest is for younger disadvantaged individuals 
(Title II-A programs for persons between 16 and 21 years of age). 



ERIC 



CO 



CO 
LU 




Differences between the Title II-A adult program and the other JTPA programs m Utah are 
shown in Figure 3.2. The roughly proportional assignment of adult Title II-A participants across 
the three program activities stands in contrast to other JTPA programs, whose participants 
received a disproportionate share of one type of employment and ti-aining assistance or another. 
For example. Title II-A youth and Title III (dislocated worker) formula programs, emphasized 
classroom training. "Other Title 11" programs (such as those funded under the state set-asides: 
3% Older Workers and 8% Vocational Education programs) emphasized on-the-job ti-aining, 
while Title III discretionary programs, targeting specific manufacturing plants and industries, 
emphasized job search assistance. 

In addition. Title II-A adult programs served a disproportionate share of individuals residing 
in rural Service Delivery Areas (SDAs) compared with other JTPA programs. As seen in Figures 
3.3 and 3.4, Utah is divided into 9 SDAs, with the population served reflecting the population 
distribution of the state at large. Four SDAs account for nearly 70% of those ser\-ed ~ all 
clustered together in the urbanized area around the state capital of Salt Uke City. In fact, 2 
SDAs - Salt Lake and Mountainland- comprise over one-half of all eligible program 
participants in the state. The remaining 30% are scattered throughout the northern Bear River 
SDA and the four large, contiguous SDAs of Uintah Basin, Central, Southwest, and Southeast 
SDAs. 

Finally, men comprised over 80% of all participants in Title III programs. In all Title II programs, 
women were more equally represented, accounting for 50% of all participants. 



Utah J TPA n- A Adult Participants 

As shown in Table 3.1, adult men and women in the Title II-A programs differed little by 
race /ethnicity. Most were white (just over 80%); about 2% were black. Hispanics and "others" 
(mainly American Indians) were the largest minority groups (about 7% and 10% respectively). 

Differences between men and women appeared in their other characteristics. The Title II-A men 
were slightly younger and less educated than the women. More of the men were between the 
ages of 22 and 29 (47% compared to 41% among women) and more of the men were school 
drop-outs (22% compared to 17% among women). 

The men showed a greater attachment to the workforce than the women. Seventy-four percent 
had been employed in the previous year, compared with two- thirds of the women. Among 
those who had been employed in this pre-entiy period, the men had earned over $5,700, while 
the women had earned considerably less ($3,251). Also, 1 in 4 of the men, but -ily 1 in 10 of the 
women, were UI claimants at the time of entiy into JTPA. Finally, a greater percentage oi women 
than of men were welfare recipients (38% versus 9%). 

The enrollment distribution across the three major activities differed between men and women 
as well. A smaller proportion of men were enrolled in classroom h-aining (23% versus 40% for 
women), while more men were enrolled in on-the-job h-aining (40% versus 35% for women) and 
also in "other activities" composed mainly of job search assistance (37% versus 2j%). 
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Figure 3.2 

JTPA Enrollee Characteristics By Program 
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Figure 3.3 
Utah Service Delivery Areas 
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Figure 3.4 
Title IIA Adult Population by SDA 
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Table 3.1 



PARTICIPANT CHARACTERISTICS OF UTAH'S 
TITLE II-A ADULTS DURING PY 1987 
(Percent Distribution) 



= — — = 


UTAH 


CHARACTERISTIC 


WOMEN 


MEN 


[NUMBERl 


19361 


[9281 


ETHNICITY: 






WHITE (Non-Hispanic) 


82.6 


80.3 


BLACK (Non-I lispanic) 


1.6 


2.3 


HISPANIC 


5.9 


7.9 


OTHER 


9.9 


9.6 


SCHOOLING : 






HIGH SCHOOL DROP-OUT 


16.5 


22.0 


HIGH SCI lOOL GRAD/GED 


51.0 


44.9 


POST-HIGH SCHOOL 


32.6 


33.1 


AGE: 






22-29 


41.3 


46.7 


30-44 


48.2 


45.6 


45-54 


10.5 


7.8 


WELFARE RECIPIENT 


35.7 


8.9 


UI CLAIMANT 


9.1 


24.5 


PRIOR YEAR: 






PERCENT EMPLOYED 


66.8 


74.1 


EARNINGS IF EMPLOYED 


3251 


5751 


NUN4BER OF FIRMS 


1.77 


1.97 


ACTIVITY (PREDOMINANT): 






CLASSROOM TRAINING 


39.6 


23.1 


ON-THE-JOB TRAINING 


35.3 


40,2 


OTHER ACTIVITIES 


25.1 


36.7 



PRIOR EMi LOYMENT AND EARNINGS EXPERIENCES OF TITLE II-A 
ADULTS 

The theory of "human capital" provides the conceptual basis for the earnings models developed 
in the study: the idea is that an individual's current employment status and earnings are 
reflections first and foremost of prior investments of human capital - including schooling, 
formal occupational training, and work experience."* In addition, the theory is broadened to 
allow for unobserved factors - including motivations and attitudes related to work, language 
barriers in the work place among others -- to affect individuals' employment and earnings. 
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The identification of "similar" comparison groups, which are used as proxies for the expected 
labor market outcomes of program participants had they not enrolled in JTPA, is critically 
dependent upon the specification of pre-entry employn^ent and earnings. Knowledge of the 
actual work history of enrollees is key. 

As seen in Figures 3.5 and 3.6 below, these experiences differed significantly between women 
and men enrolled in Utah's PY 1987 Title II- A adult programs. As shown in Figure 3,5, both the 
levels and trends of employment rates leading up to enrollment differ significantly between the 
two gender groups. The quarterly employment rates of adult men averaged nearly 10 
percentage points higher than adult women, with the differential eroding closer to the 
enrollment date. The proportion of adult men who were employed declined over time, while 
adult women were more likely to be employed closer to the enrollment date. In addition, those 
enrolled in the more intensive training programs (i.e., classroom training and OJT) had higher 
employment rates than those assigned to other activities. 

As seen in Figure 3.6, of those employed, the quarterly earnings differences between the two 
gender groups differed as well. Adult men's quarterly earnings were significantly higher than 
the women's in the earlier pre-entry period and were subject to a greater temporary decline the 
closer to the JTP A enrollment period. Specifically, employed adult men earned over $3,000 per 
quarter 8 quarters prior to entry and experienced over a 50% decline in earnings over the two 
year pre-entry period. The decline in earnings for adult women over this period was less 
dramatic, falling roughly one-third from an initial level of $1,500 per quarter. 

The patterns of pre-entry earnings over this period for those enrolled in OJT also differed 
between women and men. The women enrolled in OJT experienced the greatest decline in 
quartorly earnings, whereas adult men enrolled in OJT experienced the smallest decline. This 
difference is important since significantly different program impacts are found for this program 
activity between the two gender groups, 

Tl^is description of personal demographics, schooling, and prior work experiences of Utah's 
Title II-A adult participants is fundamental to the derivation of "similar" comparison groups 
and the estimation of JTPA program impacts in the chapters that follow. In the next chapter 
these participant characteristics will be used to identify alternative comparison groups, which 
will be used to derive estimates for JTPA program impacts for Utah's adult Title II-A programs. 
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Figure 3.5 
Pre-Entry Percent Employed 
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Figure 3.6 
Pre-Entry Earnings 11 Employed 
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Endnotes 



1 . Ovc r 5% of all entrants during PY 1987 had initially enrolled in the previous program year 
and were either still enrolled in their initial activity or had been re-enrolled in a separate activity 
during PY1987. 

2. It may be noted that the departure point used in the study is the entry date as opposed to 
the exit date commonly used in JTPA reports. The difference reflects the nature of net impact 
analysis used in the study. In particular, the treatment group enrolling in JTPA is compared to 
a "similar" comparison group of individuals who choose not to enroll but are available for 
full-time employment at the time of interruption in (or entry to) their employment history. As 
such, an important "cost" to program participants is the foregone earnings they could have 
realized had they not enrolled. Thus the appropriate employment interruption period begins 
with program entry not exit. 

3. The study omits any individual who was previously enrolled in a Title II-A adult program 
in an earlier program year to ensure "pre-ontry" does not include time a participant spent in the 
program rather than in the work force. In addition, the study excludes program year "repeaters" 
(composing less than 3% of all participants) to ensure pre- or post-training quarters are not 
contaminated with program participation. Utah state JTPA data records alone do not identify 
such individuals, since program operators utilize the same resources (except perhaps intake 
resources) regardless of whether one participant undertakes two separate services or two 
participants undertake one service each. 

4. This theory, first popularized by Becker (1964) and later tested empirically by Mincer (1974), 
has since become standard in most applied labor economics literature. 

5. This "dip" in pre-entry earr/ngs was first reported by Ashenfelter (1978) and Kiefer (1979) 
for economically disadvantaged adults under Manpower Development Training 
Administration (MDTA) in the 1960s and later documented for similar individuals served under 
CETA. See, for example: Bassi (1984), Ashenfelter and Card (1985), and LaLonde (1986). It may 
be noted that this pattern of pre-entry earnings was reported by the National Commission for 
Employment Policy (1992) for over 60,000 economically disadvantaged adults pooled across 11 
states. 
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CHAPTER FOUR: JTPA il-A Adult 
Comparison Group Selection 



As discussed earlier, the methodology used to evaluate the effectiveness of employment and 
training programs is a comparison of the actual post-training outcomes of participants against 
"counter factual" outcomes - those outcomes that hypothetically would have occurred had 
cnrollecs not participated in JTPA. With the non-experimental approach taken in this study, 
the counterfactual outcomes are approximated by a select comparison group which should be 
similar to the JTPA treatment group - for both observed and unobserved factors - except for 
program participation. The objective of this chapter is to identify such a comparison group. 

The identification of a comparison group that is truly representative of JTPA cnrollees is difficult 
for two reasons. First, one must find a large enough sample of individuals who arc eligible for 
enrollment and are distributed similarly to JTPA enrollees with respect to observed 
demographics and employment histories. Second, unobserved factors which affect the decision 
to enroll in JTPA and are related to subsequent earnings and employment should be distributed 
similarly between the enrollees and the comparison group. In practice, one attempts to select 
individuals who are similar to JTPA enrollees with respect to major observed characteristics; 
unobserved differences must be controlled by statistical modeling. 

The discussion explaining how a comparison group is selected in this non-experimental study 
is divided into three major parts. First, the procedure used to identify alternative comparison 
groups from a random sample of Employment Security (ES) registrants is given. The initial data 
base is described; then the methodology for choosing comparison groups is explained, followed 
by a description of three alternative comparison groups selected for consideration in the 
non-cxpcrimcntal impact models developed later. 

The second part of the chapter describes in non-mathematical terms how one may choose the 
"most similar" comparison group for use in program impact analysis. This choice is guided by 
specification tests which arc applied to each group to indicate the presence and extent of 
selection bias. First, a scries of tests arc proposed to examine if selection bias can be removed 
by modeling the JTPA selection process with observed factors; next tests are proposed to 
determine if the bias can be removed by modeling the JTPA selection process with unobserved 
factors using more complex statistical techniques. 

The last part of the chapter subjects the alterative comparison groups of Utah ES clients to the 
selection bias tests and identifies the "most similar" comparison group. This group is used in 
the following chapter for estimating unbiased program employment and earnings impacts of 
JTPA training. 
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PART I; Identifying Comparison Groups For Non-Experimental 
Evaluations Of JTPA 



DATA BASE FOR IDENTIFYING COMPARISON GROUP 

The comparison groups used in this study are drawn from a sample of ES registrants who 
received basic employment assistance during PY1987. Individuals processed through Utah's 
Employment Security agencies are eligible for job search assistance - including assessment, 
counseling, resume wnting preparation, and job placement with listed vacancies. These are 
common forms of basic employment assistance JTPA enrollees could expect to receive if they 
had not instead enrolled in more intensive government-subsidized training programs? The 
sample of ES registrants chosen is a 10% randomly chosen sub-set, which was designed to select 
approximately 10,000 individuals from whom a comparison group could be identified. This 
procedure was used as a second best approach to selecting only those individuals who were 
coded "economically disadvantaged.'*^ 

From the original total of 9,836 Employment Security applicants in the 10% sample, a sub-set of 
5,851 observations was selected who met the age eligibility criterion (22 to 54 years of age at 
entry to the Title Il-A adult program) and whose education, gender, and race were known."* 
These records were then matched with another ES record containing the application date and 
finally merged with Utah UI wage records. This process produced 5,413 complete records 
containing the intake information along with 16 quarters of earnings records — 8 pre-ES and 8 
post-ES application. 

As can be seen in Table 4.1, the resulting ES sample differs significantly with respect to observed 
personal characteristics and employment histories from that of the 1,864 JTPA II-A adults who 
enrolled for the first time during PY 1987. The randomly chosen ES registrants are composed 
of greater proportions of whites, males, and more highly educated individuals. They are less 
likely to be in the prime of their work history (aged 31 through 40), and more likely to be older 
(aged 41 through 54). 

Two of the most salient differences betv/een the two groups are residential location and welfare 
status. ES clients are far less likely to be on welfare (1.47o versus 22.37o of JTPA participants), 
and are far more likely to be living in urbanized Wasatch Front South and in Weber-Morgan 
and Davis SDAs. This differential mainly reflects the state JTPA allocation formula, which 
places great weights on unemployment rates and low wages — both of which favor the less 
densely populated rural areas of Utah. 

In addition, ES clients register evenly throughout the p jgram year, whereas JTPA enrollment 
is partially influenced by the timing of program year mnding. For example, a greater portion 
is enrolled in JTPA in the first quarter of the program year when funding is plentiful and fewer 
are enrolled during the last quarter when the year's budget is low. 

The distribution of ES clients by the length of continuous spells of unemployment also differs 
from that of JTPA enrollees. While JTPA enrollees are more likely to experience intermediate 
spells of unemployment prior to entry, ES clients are more likely to experience either no 
unemployment just prior to entry or they tend to be completely out of the labor force during 
the 2-year period prior to intervention. While the random sample of ES clients' industry 
distribution (2 years prior to entry) is similar to that of their JTPA counterparts, they are either 
more likely to be fully attached to the labor force (work all 4 quarters) or cut of the labor force 
entirely (zero quarters of covered employment).^ These differences exist for both of the 
pre-enrollment years and are greatest in the year just prior to entry. JTPA II-A adults, on the 
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TABLE 4.1 



BACKGROUND CHARACTERISTICS OF JTPA II-A AD I ^T PROGRAM AND 
ES REGISTRANT ADULT SAMPLE (Percent Distribution) 



CHARACTERISTIC 


JTPA II-A 
ADULT 


EMPLOYMENT 
SECURITY 


[NUMBER] 


[1,864] 


[5/413] 


GENDER: FEMALE 


49.9 


43.3 


MALE 


50.1 


56.7 


ETHNICITY: WHITE (NOW -HISPANIC) 


81.3 


89.1 


OTHER MINORITIES 


19.4 


10.9 


SCHOOLING: H.S.DROP-OUT 


19.6 


13.4 


H.S.GRAD/GED 


48.4 


42.8 


POST-H.S. 


32.0 


43.8 


AGE: 22-25 


24.2 


26.0 


26-30 


24.5 




31-35 


21.5 


176 


36-40 


14.2 


12.5 


41 + 


15.6 


18.6 


RESIDENTIAL LOCATION: URBAN 


56.5 


72.3 


RURAL 


43.5 


27.7 


PCT. WELFARE RECIPIENTS 


22.3 


1.4 


ENTRY QUARTER: 87:3 


36.6 


24.7 1 


87:4 


22.6 


22.9 j 


88:1 


27.7 


26.9 j 


88:2 


13.1 


25.5 j 


QUARTERS OF CONTINUOUS SPELLS OF UNEMPLOYMENT: NONE 


45.6 


56.6 j 


ONE-TWO 


17.7 


8.9 j 


THREE-SEVEN 


14.3 


8.1 ! 


EIGHT OR MORE 


22.4 


26.3 


INDUSTRY AFFILIATION IN 2ND YEAR PRIOR TO INTERVENTION: 






CONSTRUCTION 


10.2 


11.0 


DURABLE MANUFACTURING 


10.6 


11.4 


NON-DURABLE MANUFACTURING 


7.1 


6.7 


WHOLESALE/RETAIL TR.^DE 


29.0 


28.8 


BUSINESS SERVICES 


28.4 


25.9 


OTHER INDUSTRIES 


14.7 


16.2 


WORK FORCE ATTACHMENT 2ND YEAR PRIOR TO INTERVENTION: 






OUT OF LABOR FORCE 


36.1 


45.5 


PARTLY ATTACHED (1-3 QRS) 


34.3 


21.4 


FULLY ATTACHED (4 QRS) 


29.6 


33.1 


WORK FORCE ATTACHMENT 1ST YEAR PRIOR TO INTER VENllON: 






OUT OF LAIK)R FORCE 


29.6 


42.9 


PARTLY ATTACI lED (1-3 QRS) 


48.5 


26.7 


FULLY ATTACHED (4 QRS) 


21.9 


30.4 
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other hand, are more likely to be partly attached to the labor force (working from 1 to 3 quarters), 
especially in the year just prior to enrollment. 



These observed differences between the JTPA and ES groups, along with possible unobser\'ed 
differences, result in considerably different pre-entry employment and earnings patterns. As 
seen in Figure 4.1, differences between the two groups with respect to their prior employment 
rates (percent of group employed at any time during a quarter) and employed earnings 
(earnings of those employed) are quite evident. The randomly chosen ES client sample has 
higher rates of employment in all quarters prior to intervention, and those employed have 
higher earnings. These latter characteristics become especially evident in the 4 quarters just 
prior to intervention due to the decline in employment of JTPA II-A adults in the year prior to 
entry. While the employment rates differ little between the 2 years for JTPA enrollees, the 
variability of their employ men t increases dramatically in the year prior to entry. In comparison, 
ES clients experience a gradual rising rate of employment over the 2-year pre-entry period. 

An even more striking difference is observed with respect to the quarterly earnings pattern of 
those employed in a given pre-entry quarter. The quarterly earnings of employed JTPA trainees 
falls nearly 50% - from $2,500 to $1,250 - while that of ES clients rises slightly from an initial 
level of $3,500. This observed dip in quarterly earnings of JTPA enrollees, as compared to ES 
clients, could result from sudden unfavorable working conditions, personal health problems, 
or the like.^ 



Methodologies Used For Choosing Comparison Groups 



The identification of a "similar" comparison group is, perhaps, the most important and difficult 
task when applying non-experimental techniques to estimate quasi-net impacts of JTPA 
training. In the past, most researchers relied on national surveys from which they attempted to 
identify a sub-set of individuals who were similar to Comprehensive Employment and Training 
Administration (CETA) and JTPA participants being analyzed. As noted in Appendix B, for 
example, the Current Population Survey (CPS) March edition was used to derive a comparison 
group for the National Supported Work (NSW) Demonstration. The wage records of these 
individuals were then matched against Social Security Administration (SSA) files and, due to 
confidentiality requirements imposed by SSA, were aggregated into defined cells according to 
spocif ic observed personal demographic characteristics. As described in Appendix B, this early 
comparison group approach had two flaws: it was not possible to identify and eliminate those 
individuals who enrolled in CETA, and the pre-enrollment employment and earnings patterns 
differed dramatically between the treatment group in NSW and the comparison groups selected 
from the CPS. In essence, the two groups were so different that neither matching procedures 
nor statistical techniques to control for observed differences were effective in deriving a "similar" 
comparison group. In addition, using the CPS-SSA data files necessitated analyzing grouped 
data, rather than individual data, due to the confidentiality restriction imposed by SSA, as noted 
above. 

Other approaches to identify compai ison groups in experimental studies using individual data 
have used control groups from different demonstration sites, or control groups from an earlier 
time period in a given demonstration site for separate non-experimental analysis. Because 
these approaches are based upon control groups from random assignment experiments, they 
can not be used for non-experimental studies which, by their nature, do not have randomly 
assigned control groups. 
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FIGURE 4.1 

PRE-ENTRY WORK HISTORY: JTPA 
& EMPLOYMENT SECURITY SAMPLE 



EMPLOYMENT RATES 

60 I 




40' ' ^ ' " 1 I 

-8 -7 -6 -5 -4 -3 -2 -1 

QUARTER FROM ENTRY 



QUARTERLY EARNINGS 



4000 




1000' ' ' ^ ^ 1 1 

-8 -7 -6 -5 -4 -3 -2 -1 

QUARTER FROM ENTRY 



JTPA EMPLOYMENT SECURITY 




For the vast majority of JTPA programs operated by the approximately 650 SDAs across the 
country, researchers wishing to evaluate JTPA as it works on a day-to-day basis must find 
proxies for the "counterf actuals" of JTPA participants. That is, they must find individuals who 
are similar to JTPA participants with respect to all observed and unobserved factors except one 
- participation in JTPA. The comparison group chosen should approximate the post-training 
experience of JTPA participants had they not enrolled in JTPA (the hypothetical counterf actual 
that by definition does not exist). 

Clearly, the 10% random sample of ES registrants differs significantly with respect to observed 
demographic factors and pre-entry employment and earnings patterns from that of JTPA II- A 
adult trainees. As such, use of the 10% sample as a comparison group would yield biased 
estimators in the quasi-net impact models of JTPA training. 

Two approaches, not mutually exclusive, have been proposed to address the selection problem 
in non-experimental studies. First, cell matching techniques can be used to screen out possible 
comparison group members who do not share a combination of observed factors similar to the 
treatment JTPA group. One technique, the "nearest-neighbor" approach, was used to evaluate 
selected CETA programs.^^ Another approach is to use regression analysis, combined with 
various assumptions about the nature of the JTPA selection process, in quasi-experimental 
studies to "control for" both observed and unobserved differences between comparison groups 
and JTPA treatment groups. These two approaches can be used together, with the comparison 
group first refined using cell matching techniques and then regression analysis used to control 
for the observed and unobserved differences between the comparison group members who 
passed through the cell matching screens and the treatment group. This latter approach is taken 
in this study. 



ALTERNATIVE COMPARISON GROUPS FOR UTAH STUDY. 

The first step in developing similar comparison groups is to apply cell matching techniques 
which will result in a sub-set of the 10% sample that more closely resembles the JTPA treatment 
group in its observed demographic characteristics and pre-enrollment employment and 
earnings histories. As shown in Table 4.2 below, three such comparison groups are considered. 

In each case, the 10% ES sample was first stratified according to four gender-race groups: white 
males, non-white males, white females, and non-white females. ^ Next, alternative cell 
matching strategies were applied as a series of screens to each stratified gender-race group to 
derive the three comparison groups. In the first group, the sub-set is reduced in half by an initial 
cell matching of the age-schooling distribution and the distribution of the change in earnings 
(Group A). 

The next two are reduced in half by matching either the change in employment distribution 
(Group B) or the change in earnings distribution (Group C) over the 2-ycar pre-registration 
period for the ES 10% sample. In both of these groups, cell matching according to observed 
personal characteristics is purposely avoided. A third and final ce!: matching technique is 
applied to the last two comparison groups with regard to the distribution of pre-entry annual 
earnings. 

For all three comparison groups, the final matching reduces the sample by an additional 15%, 
resulting in comparison groups nearly 70% smaller than the original sample and roughly equal 
in number to the JTPA li-A adult participant group. 
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TABLE 4.2 



REDUCTION IN ORIGINAL SAMPLE SIZE DUE TO 
SELECTIVE COMPARISON GROUP SCREENS 



SCREEN 


COMPARISON GROUP CELL SIZE ' °] 


GROUP A 


GROUP B 


GROUP C 1 


ORIGINAL SAMPLE 


5413 


5413 


5413 


SCREEN # ONE: 

Age-schooling Groups 


3708 (-31.5%) 






SCREEN # TWO: 

Change In Employment 
Change In Eam'mgs 


2557 (-21.3%) 


2467 (-54.4%) 


2527 (-53.3%) 


SCREEN # THREE: 
Earnings (2nd Year) 
Earnings (1st Year) 


1716 (-155%) 


1659 (-15.0%) 


1635 (-16.5%) 


(TOTAL PERCENT REDUCTION) 


-68.3% 


-69.0% 


-69.8% 



The first cell matching described above results in a comparison group iGroup A) selected to 
replicate the age and schooling distribution of JTP A enrollees as closely as possible. To the 
extent possible, additional cell matching is applied to this sub-set ir order to replicate the level 
of earnings in^the second year prior to entry, as well as the fall in pre-entry earnings in the year 
prior to entry. This comparison group may be viewed as one that assumes the observed forms 
of human capital are the important characteristics of a comparison group and should therefore 
be matched as closely as possible to the levels found in the JTPA enrollee group. Two major 
forms of human capital investments - schooling and general work experience, as approximated 
by age - are used in this matching stage. The second year employment and earnings focus of 
this sub-set is based upon the assumption that the best measure of a worker's stock of human 
capital and earnings potential is that existing before the year just prior to enrollment. Measures 
of work experience in the year just prior to JTPA enrollment may understate the trainee's true 
stock of human capital due to the pre-enrollment dip in earnings that characterize employment 
and training participants; a similar understatement would result if the focus on quarteriy 
employment and earnings of ES registrants were on the year just prior to entry. 

Two additional comparison groups are chosen which ignore the distributions of age and 
schooling as measures of human capital, and emphasize instead the unobserved factors that 
give rise to the dip in pre-entry employment (Group B) and earnings (Group C) not explained 
by human capital factors. The ceil matching strategy in the second comparison group (Group 
B) first replicates the JTPA distribution of employment in the second year prior to entry 
according to a worker being out of labor force, partly attached, or fully attached to the labor 
force. Next, the distribution of the change in employment status from the second to the first 
year prior to entry of JTPA trainees is replicated for each of the three employment categories. 
To the extent possible, final cell matching is applied to the distribution of quarterly earnings in 
the year prior to entry so as to approximate the observed dip in earnings of JTPA enrollees.^"^ 
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A final, round of matching is applied to the third comparison group (Group C), which 
emphasizes the change in quarterly earnings over the 2-year pre-entry period. Like the second 
comparison group above, the cell matching technique is applied to the JT?A participant 
distribuHon of employment in the second year prior to entry --^whether a worker is out of labor 
force, partly attached, or fully attached to the labor force. Next, cell matching is used to 
duplicate the JTPA distribuHon of the change in quarterly earnings from the second to first year 
prior to entry. A final cell matching is applied to the distribuHon of quarterly earnings in the 
year prior to entry to approximate the observed dip in earnings of JTPA enrollees. 

In all cases, the number of observations in each comparison group is constrained by attempHng 
to maintain a one-to-one raHo between the comparison and treatment groups for the four 
gender-race categories. As shown in Table 4.3, the varying cell matching techniques applied to 
the 10% ES sample for each comparison group result in differing distribuHons of demographic 
characteristics, pre-entry employment and earnings patterns, and industry affiliaHon relaHve to 
the JTPA treatment group. 

Group A has similar distribuHons of demographic characterisHcs, whereas the other two groups 
have a greater proportion of males and are more highly educated and younger (i.e., more are 
25 years of age or younger) compared with JTPA parHcipants. In addiHon, it appears the cell 
matching techniques used to derive the three comparison groups result in small but important 
differences with regard to prior work histories. In the year prior to intervenrion, the ES 
comparison groups chosen on the basis of earnings changes (Groups A and C) were 50% more 
likely to be employed all 4 quarters (35% and 32% versus 20%) than the comparison group 
(Group B) chosen to simulate the employment status change of the JTPA group. 

The pre-intervcntion pattern of quarterly employment and earnings (if employed) of the 
comparison groups relaHve to JTPA enrollees are shown in Figure 4.2. While the two 
comparison groups matched on pre-entry quarteriy earnings (Groups A and C) display rising 
employment rates prior to ES intervenHon, the second comparison group - which simulated 
the change in employment of JTPA enrollees (Group B) - displays an employment rate pattern 
similar to the JTPA group. All three comparison groups display the fall in quarterly earnings 
of those employed. The two groups which simulated the earnings change r-^nerns (Groups A 
and C) differ from the second comparison group (Group B) by having relaHvely high earnings 
in the second year prior to entry. In addiHon, the first comparison group (Group A), which 
replicates the observed demographic and schooling distributions of JTPA enrollees, has a flatter 
earnings decline from its relatively high original level of quarteriy er.mings. 

In summary, the search for a comparison group similar to JTPA II-A adult program enrollees 
during PY 1987 using the 10% sample of ES parHcipants with cell matching techniques results 
in differing types of comparison groups. On the one hand, comparison Group A seems to be 
more similar to JTPA enrollees with respect to observed personal demographic factors, but 
relatively different from JTPA enrollees with respect to pre-entry quarterly earnings and 
employment. On the other hand, comp .rison Groups B and C show a greater dip in pre-entry 
earnings and employment patterns (especially Group B) - which may be related to unobserved 
factors causing a temporarily worsening economic condition - but have a different distribuHon 
of observed demographic factors than Group A. 

It appears that cell matching that more closely replicates observed human capital variables is 
not related to unobserved factors which could produce the dip in pro-entry employment and 
earnings of JTPA enrollees. Similarly, cell matching that more closely replicates the dip in 
pre-entry earnings is les3 similar to JTPA enrollees with respect to observed human capital 
variables. There do not appear to be cell matching techniques, using these data, which 
incorporate both observed and unobserved factors in a way to generate human capital 
distribuHons and pre-entry employment and wage patterns similar to that of JTPA enrollees. 
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TABLE 4.3 



BACKGROUND CHARACTERISTICS OF JTPA II-A ADULT PROGRAM AND 
ES REGISTRANT ADULT SAMPLE (Percent Distribution) 



CHARACTERISTIC 


JTPA 


cr.oup A 


GROUP B 


GROUP C j 


[NUMBER] 


[1,864] 


[i,6ic: 


[1,573] 


[1,530] j 


GENDER: FEMALE 


49.9 


48.8 


47.9 


47.1 


MALE 


50.1 


51.2 


47.9 


47.1 


ETHNICITY: WHITE (NON -HISPANIC) 


81.3 


81.1 


81.1 


81.8 


OTHER MINORITIES 


19.4 


18.9 


18.9 


18.2 


SCHOOLING: H.S.DROP-OUT 


19.6 


18.4 


16.3 


15.4 


HS.GRAD/GED 


48.4 


50.1 


41.6 


41.9 


POST-H.S. 


32.0 


31.5 


42.0 


42.5 


AGE: 22-25 


24.2 


25.3 


29.4 


30.5 


26-30 


24.2 


7.72 


7.6 


25.9 


31-35 


21.5 


18.2 


16.8 


16.1 


36-40 


14.2 


13.0 


10.6 


11.8 


41+ 


15.6 


15.8 


15.6 


15.8 


RESIDENTIAL LOCATION: URBAN 


56.5 


39.8 


42.3 


42.4 1 


RURAL 


43.5 


60.2 


57.7 


57.6 


per. WELFARE RECIPIENTS 


22.3 


1.0 


1.4 


1.2 


ENTRY QUARTER: 873 


36.6 


25.7 


25.3 


24.2 


O/ .*t 


ZZ.D 




ZZ.vJ 


01 0 
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OO. 1 
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07 
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0^ ^ 
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1 
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NON-DURABLE MANUFACTURING 


7.1 


4.8 


4.6 


4.9 


WHOLESALE/RETAIL TRADE 


29.0 


11.4 


10.0 


10.8 


BUSINESS SERVICES 


28.4 


35.5 


33.2 


32.6 


OTHER INDUSTRIES 


14.7 


32.3 


37.0 


35.7 


WORK FORCE ATTACHMENT 2ND YEAR PRIOR TO 
INTERVENTION: 










OUT OF LABOR FORCE 


36.1 


32.3 


37.0 


35.7 


PARTLY ATTACHED (1-3 QRS) 


34.3 


35.8 


33.6 


33.7 


FULLY ATTACHED (4 QRS) 


29.6 


31.9 


29.4 


30.6 


WORK FORCE ATTACHMENT 1ST YEAR PRIOR TO 
INTERVENTION: 
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FIGURE 4.2 

PRE-ENTRY WORK HISTORY: 
JTPA AND COMPARISON GROUPS 
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Given this situation, it is interesting to note the distributions of pre-entry employment and 
earnings of the comparison group members who classified themselves "economically disadvan- 
taged" on the self-reporting portion of the ES intake form. Figure 4.3 shows the distribution of 
the sub-sets of economically disadvantaged within each of the three comparison groups (they 
comprise roughly 1 3%, or nearly 225 observations, of each group). The economically disadvan- 
taged have significantly lower employment rates and quarterly earnings, especially in the 
second year prior to entry into the ES system. From these observed patterns of employment 
and earnings, it appears that those declaring themselves as being "economically disadvantaged" 
are truly disadvantaged. In the future, it may prove worthwhile to choose a comparison group 
for JTPA from the population of those who report themselves as being disadvantaged. 



PART II: Tests For Selection Bias In Non-Experimental Models 



The objective of deriving a comparison group similar to the JTPA trainee group is to reduce the 
likelihood of obtaining an estimate of JTPA training that is "biased" due to the selection process. 
A prorrram impact estimate of JTPA participation is said to be biased if omitted factors related 
to se:c'Ction into JTPA are related to the post-training employment and earnings outcomes. 

For example, consider a situation where individuals who are less motivated to work, who have 
greater learning disabilities, or who have poorer health are less likely to search for and find 
employment through the Utah ES, In time, they turn to JTPA for employment assistance in 
hopes of finding a job. Following JTPA training they find employment in relatively less stable 
and lower paying jobs that are matched to their skill levels, health, and work habits. If the 
post-training employment and earnings pattern of these individuals are compared with 
comparison group members who have stronger motivation to work, fewer learning disabilities, 
and better health - the results will be "biased" against the JTPA trainee group,^'' Information 
regarding the selection process into JTPA is needed to "control for" the impact of these factors 
on post-training employment. The problem, however, is that both observed and unobserved 
factors may affect the selection process. If relevant factors are excluded from the earnings 
models which are systematically related to the selection into JTPA, "selection bias" will result. 

To remove the selection bias it is necessary to derive a "similar" comparison group using as many 
related factors as possible so as to reduce the number of factors which, if omitted, would bias 
the program impact estimate. Then particular modeling specifications are tested to control for 
observed and unobserved factors that may further bias the program impact estimates. 

Since an early evaluation of MDTA by Ashenfelter in 1978, it has been evident that 
non-experimental approaches to estimating training effects need to be subjected to various tests 
to ensure the validity of the comparison groups as good proxies for the treatment counterfactual, 
and thus eliminate selection bias. It was not until recently, however, that economists developed 
more thorough selection bias tests of comparison groups used in the evaluation of training 
programs. 

In 1989, Heckman and Hotz developed a series of tests which compared both pre-enrollment 
and post-enrollment earnings of experimental control groups and selected comparison groups 
for the National Supported Work (NSW) demonstration. The objective of their study was to 
determine if the wide range of training estimates produced by non-experimental models — and 
attributed to selection bias — could be reduced by identifying only those comparison groups 
which could pass more stringent validity tests,^^ 
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FIGURE 4.3 

JTPA II-A Adults & Economically Disadvantaged: 
Prior Employment and Earnings Pattern 
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The selection bias tests used here rest on the assumption that the earnings generating process 
existing prior to government intervention is similar to that following intervention, and ~ once 
both observed and unobserved factors of the selection process are controlled - any remaining 
difference between the post-training earnings of the treatment and comparison groups must be 
due to differences in the intervention strategies (i.e., JTPA versus ES). 

The selection bias tests for the Utah JTPA study were constructed to test the null hypothesis that 
selection into JTPA has no statistically significant effect on earnings before JTPA training is 
received. If the two groups already differ sigi\ificantly prior to treatment, this suggests that 
post-training differences are not solely due to the difference in services received. In all tests, a 
binary training participation variable (1 = yes; 0 = no) was added to pre-entry earnings models 
along with factors related to the selection process. The expectation was that the training 
coefficient would not be significantly different from zero. If, for example, the JTPA coefficient 
in the pre-entry earnings model differs from zero (i,e., the model specification "fails the test"), 
one can expect the model specification to yield biased estimates of post-training earnings 
impacts. If, however, the JTPA coefficient in the pre-entry earnings model does not differ from 
zero significantly (i.e, the model specification "passes the test"), use of the particular earnings 
specification for the post-training period would yield unbiased estimates of post-training 
earnings impacts. 



Measure Of Pre-Intervention Earnings For Selection Tests 



The selection bias tests performed in this study are based on pre-entry earnings measures which, 
unfortunately, were constrained by available data. Ideally, one would like to use several years 
of pre-entry earnings so as to fully specify the pre-entry earnings process. In the case of the 
Utah study, only 2 years of pre-enrollmen t data were available. The transitory nature of ean ' ngs 
in the year prior to entry into JTPA makes that year's earnings a poor measure of the earnings 
potential of the treatment group. Furthermore, using the second year of earnings prior to entry 
in the selection bias tests is problematic as no earnings prior to this period can be specified in 
the earnings models - a characteristic critical to the longitudinal nature of earnings models. 
As such, the measure of pre-entry earnings used in all selection bias tests below is the sum of 
earnings in seventh through twelfth month (i.e. the third and fourth pre-entry quarter) prior to 
entry into JTPA or registration for ES services. This is the closest period before the dip in 
earnings experienced by nnany JTPA participants prior to enrollment. It is also a 2- quarter 
period that can be compared with the same 2 calendar quarters in the second year prior to entry 
(i.e., seventh and eighth pre-entry quarters) available in the data set. In this way, the earnings 
measured over common calendar quarters will not be affected by the seasonality present m 
quarterly eami ags. In essence, the two 6-month earnings periods are surrogates for two earlier 
pre-entry annual earnings periods that would have been used had they been available for this 
study. 



Selection Bias Tests Developed For Utah Study 



Five selection bias tests are conducted for each of the Utah comparison groups derived above, 
and are estimated for women and men separately. The first three test-models of the selection 
process use only observed factors, while the next two test-models incorporate unobserved 
factors in the JTPA selection process. Each of these tests makes different assumptions about the 
dependence between the training variable and the error term of pre-enrollment earnings. The 
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model specifications were chosen to represent various ways to remove the dependency between 
JTPA selection and the pre-entry earnings, thereby eliminating any suspected selection bias. 



S election Bias Tests On Observed Factors. In the first test, only the cell matching techniques 
on observed factors were used to define the selection process. Then a univariate regression 
model which specifies only the JTPA training binary variable was used to test the null hypothesis 
of no difference pre-training earnings. If the test is rejected (i.e., the coefficient is significantly 
different from zero), one may conclude that use of the cell matching technique alone would 
likely result in biased estimates of post-training earnings impacts. 

The second model uses a simple "linear control function" which adds observed factors relevant 
to the selection process to the regression model mentioned above in addition to the JTPA training 
binary variable. If the coefficient on the JTPA variable is still significant, selection bias continues 
to characterize non-experimental models of post-training earnings even after the available 
observed factors are included. 

The third technique is called the "two stage least squares" (2SLS) method, which was proposed 
initially by Bamow, Cain and Goldberger (1980) for use in evaluating employment and training 
programs. In the first stage, the probability of enrollment was modeled and the estimate of the 
enrollment probability Oi trainees was then used in the second stage earnings model rather than 
the actual training binary variable. If all factors affecting both the selection decision and 
earnings are specified, and if the decision to enroll is a linear function of all explanatory 
variables, the dependency between the (estimated) training variable and the error term is 
eliminated. Under these conditions, estimates of post-training earnings impacts estimated with 
the 2SLS model would yield unbiased estimators. 

Selection Bias Tests on Unobserved Factors ^ A second group of validity tests was developed 
based on models that incorporate unobserved factors to the JTPA selection process. The first 
version, as used by Bassi (1983), is the "fixed effects" model which removes the selection bias if 
unobserved factors which affect the selection process and the level of pre-entry earnings are 
constant over time. Put another way, by transforming the dependent earnings variable into the 
change of earnings, selection bias can be eliminated if the impact of unobserved factors on the 
level of earnings is the same for each year. (If more motivated individuals make $1,000 more 
each year, then the impact of being more motivated on the change in earnings is zero.) In this 
form, the dependence between the error term of the newly specified dependent variable and 
JTPA training variable is removed and fixed effects model estimates of post-training earnings 
would be unbiased. 

The second version removes the selection bias induced by unobserved factors affecting 
enrollment and post-training earnings, as developed originally by Heckman (1979). This 
technique models the selection process simultaneously with post-training earnings by using the 
error structure of the enrollment model as an explanatory variable in the earnings equation. 
This non-linear simultaneous equation model version will yield unbiased estimates if its 
assumptions concerning the structure of errors are valid.^ 



PART III; Results Of Selection Bias Tests For Utah Study 



The five selection bias tests described above were applied to the 6-month pre-entry earnings of 
adult women and men who enrolled in JTPA for the first time during PY 1987 along with 
comparison group members who registered wi th ES during the same period.^^ As noted in Table 
4.4 below, all models which specify the selection process in terms of only observed factors failed 
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TABLE 4.4 



SELECTION BIAS TESTS ON PRE-INTERVENTION EARNINGS USING 
ALTERNATIVE COMPARISON GROUPS AND PY1987 JTPA II-A ADULTS: 



(A) WOMEN 



JTPA SELECTION PROCESS 


COEFFIQENT AND STANDARD ERROR OF JTPA 
VARIABLE FOR PRE-ENTRY EARNINGS MODELS: 
(Indication of Selection Bias) 


GROUP A 


GROUP B 


GROUP C 


OBSERVED FACTORS: 








Cell Matching Technique 


-242**'^ (53) 


-105** (50) 


-130** (75) 


Linear Control Function Model 


(86) 


+69** (48) 


+82** (37) 


UNOBSERVED FACTORS: 








Two-stage-least-squares Model 


-212* (125) 


-144** (72) 


-269* (152) 


Fixed Effects Model 


-124'^* (52) 


+26 (17) 


+67 (40) 


Heckman Selection Model 


-66* (38) 


+43 (45) 


+45 (62) 



( B) MEN 



JTPA SELECTION PROCESS 


COEFFICIENT AND STANDARD ERROR OF JTPA 
VARIABLE FOR PRE-ENTRY EARNINGS MODELS: 
(Indication of Selection Bias) 


GROUP A 


GROUP B 


GROUP C 


OBSERVED FACTORS: 








Cell Matching Technique 


-642*** (104) 


-213** (98) 


-278*** (97) 


Linear Control Function Model 


-261*** (66) 


-159*** (70) 


-188** (89) 


UNOBSERVED FACTORS: 








Two-stage-least-squares Model 


-1409** (649) 


-361*** (124) 


-438** (204) 


Fixed Effects Model 


-263*** (94) 


-184** (86) 


-258*** (98) 


Heckman Selection Model 


-301*** (80) 


-140* (77) 


-76 (50) 



NOTE: The level of confidence for each coefficient is indicated by the asterisks following each coefficient (* = 90%; ** = 95%; 
99%). 



the selection bias tests for both women and men across all three comparison groups. (That is, 
the coefficient on the JTPA variable v^as significantly different from zero.) 

The degree of selection bias remaining in models which include only observed factors appears 
to be greater for men than women. The test results based on cell matching alone indicate that 
women who enrolled in JTPA earned $242 less over the 6-month pre-intervention period than 
those in the first ES comparison group (Group A); men who enrolled in JTPA earned $642 less 
over this period compared w4th their counterparts in the comparison (Group A). 
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Significant improvements in the results of the selection bias tests were found when unobserved 
factors (along v/ith observed factors) were added to the pre-entry earnings models. The fixed 
effects model passed the selection bias tost (i.e,, the coefficient on the JTPA variable is not 
significantly different from zero) for the two comparison groups (Group B and Group C) that 
emphasized patterns of pre-enrollment employment and earnings in the cell matching phase 
for women; these same models failed the selection bias tests for men. The Heckman selection 
model passed the selection bias test for both women and men when using the third comparison 
group (Group C). Women who enrolled in JTPA earned $45 more than women in the third 
comparison group during the 6-month pre-intervention period, after accounting for both 
observed and unobserved factors in the pre-intervention earnings model. This difference, 
however, is not statistically significant ~ indicating increased confidence that estimates of the 
post-training impact of JTPA enrollment based on the Heckman selection model may be 
unbiased. Similarly, men who enroll in JTPA earned $76 less than men in the third comparison 
group during the pre-intervention period, but this is not statistically significant. 

In summary, the selection bias test results from the Utah study indicate that selection into JTPA 
— if not properly modeled - will result in biased estimates of program impacts, (2uasi-net 
impact estimates would be expected to vary greatly across earnings models that failed to pass 
the pre-intervention earnings selection bias test. Use of the Heckman selection model, which 
utilized information in the structure of errors in the selection process to capture unobserved 
factors, increases the confidence in post-training impact estimates may be unbiased. For this 
reason, only the Heckman selection model will be used in the Utah study to estimate the program 
impacts for Title II-A adult women and men. 



Tests For Secondary Selection Bias Among JTPA Paiticipants 



While the primary focus of the Utah study is to estimate the JTPA impact on post-training 
employment and earnings for the average enrollee, the study also investigates the differential 
impacts of participants enrolled in one of three major program activities — occupational 
classroom training (OCT), on-the-job training (OJT), and job search assistance (JSA) in the 
absence of more intensive classroom or work place training. In addition, the study also estimates 
the differential impacts on participants who complete training (compared with program 
drop-outs), and those who are placed with private sector firms following training.^'^ In this way, 
the study will go beyond determining if, on average, Utah's JTPA il-A adult program makes 
enrollees better off; it will address the next level of concern relating to what type of individual 
or program has higher (or lower) pay offs compared to the average participant. 

Given this added level of analysis, it must also be recognized that a second level of selection 
bias, not previously discussed, may characterize JTPA cnrolloes. This form of selection bias may 
occur within the JTPA treatment group once participants have selected into the program. Two 
types of secondary selection bias are considered in this study — program activity selection and 
program completion selection. 

The first type of selection occurs when participants arc sorted into available activities (or 
sequence of activities) according to personal preferences and /or assignment by the assessment 
staff of the SDA or service provider. For example, participants who are placed into — or 
self-select into — the OJT program may be more job ready and more highly motivated than those 
who enroll in classroom training or job search assistance. ^ These same individuals may be more 
likely to be employed - and if employed to earn higher earnings — than other JTPA enrollees. 
Failure to account for this secondary form of selection will result in biased estimates of program 
activity impacts.^^ 
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A second type of selection occurs when participants choose whether or not to con\plete their 
assigned activity. Participants are fully aware of the foregone earnings possible had they not 
enrolled in JTPA, and at any time participants know they could leave training for available 
employment. The decision to complete the program, as well as to be placed upon completion, 
is of critical importance to understanding why the JTPA program benefits some more than 
others. 

In both cases, failure to address the secondary forms of selection within the JTPA program may 
result in biased estimates of the impacts of JTPA program activities and program completion 
on the post-training employment and earnings of participants. Like before, various 
specifications of the earnings process can be tested for the existence of these secondary forms of 
selection bias. Since only the third ES comparison group (Group C) passed the selection bias 
tests for the average JTPA participant, only the one comparison group is used in the following 
secondary forms of selection bias tests. 

As can be seen in Figures 4.4 and 4.5, the pre-intervention employment and earnings pattern of 
JTPA participants by activity and termination status is relatively similar. Both employment 
rates and employed earnings of those enrolled in the more intensive training programs (OCT 
and OJT) track each other closely, while those enrolled in JSA-only experienced slightly lower 
employment rates and higher employed earnings. 

The employment and earnings patterns of individuals who either fail to complete the program 
or fail to be placed upon completion are also very similar, while those placed experienced 
slightly higher employment rates and employed earnings in the two year period prior to JTPA 
enrollment. 

As shown in Table 4.5, the secondary jrms of selection bias were only tested with the first two 
tests utilizing observed factors as discussed earlier. Cell matching strategies on observed 
characteristics appear sufficient to capture the selection decisions by both activity type and 
program termination status for women. In no case was the JTPA binary variable statistically 
significant for women. This finding was further supported by the linear control function test, 
which specified observed factors in the pre-enrollment earnings models for women. 

For men, the cell matching strategies would yield biased results of activity program impacts for 
those enrolling in OCT programs. Failure to control for this selection bias would result in 
activity impact estimates that would be biased upwards. This selection bias can easily be 
removed with observed factors (i.e., personal demographics, schooling, and prior employment 
and earnings patterns) in the post-training employment and earnings impact models. 

Similarly, male JTPA enrollees are characterized by secondary selection bias regarding their 
termination status. Estimates of JTPA training for those placed would be biased upwards and 
those of enrollees who completed, but who were not placed, would be biased downward. Once 
again, simple addition of observed factors to the post-training employment and earnings impact 
models appears all that is necessary to remove this form of secondary selection bias. 

In summary, selection bias tests performed on the decision to enroll in JTPA, and once enrolled 
to undertake and complete an assigned activity, have been used to identify appropriate 
specifications of models used to derive estimates of JTPA quasi-net impacts that are least likely 
to be unbiased. These tests showed that the decision to participate in JTPA is affected by both 
observed and unobserved factors that must be included in post-training employment and 
earnings models to yield unbiased estimators. In addition, the secondary forms of selection bias 
regarding the selection of an activity type and of completing the assigned activity can be 
modeled by using available observed personal demographics and pre-entry work histories used 
in the primary form of selection bias. 
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FIGURE 4.5 

PRE-ENTRY WORK HISTORY: 
JTPA TERMINATION STATUS 
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TABLE 4.5 



SELECTION BIAS TESTS ON PRE-INTERVENTION EARNINGS: 
PYly87 JTPA II-A ADULTS BY ACTIVITY AND TERMINATION STATUS 



(A) ACTIVITY TYPE 



JTPA SELECTION PROCESS 


COEFFICIENT AND STANDARD ERROR OF 
ACTIVITY VARIABLE FOR PRE-ENTRY EARNINGS 
MODELS: 
(Indication of Selection Bias) 


OCT 


OJT 


JSA 


WOMEN: 








Cell Matching Technique* 


+68 (64) 


-102 (66) 


+37 (73) 


Linear Control Function Model 


+51 (45) 


-127 (78) 


+97 (55) 


MEN: 








Cell Matching Technique 


+356** (150) 


-126 (129) 


-141 (80) 


Linear Control Function Model 


+88 (111) 


+20 (96) 


-91 (98) 1 



(B) TERMINATION STATUS 



JTPA SELECTION PROCESS 


COEFFICIENT AND STANDARD ERROR OF | 
TERMINATION STATUS VARIABLE FOR PRE-ENTRY 
EARNINGS MODELS; 
(Indication of Selection Bias) 


PLACED 


NOT PLACED 


NOT COMPLETE 


WOMEN: 








Cell Matching Technique* 


+108 (72) 


-126 (144) 


-86 (77) 


Linear Control Function Model 


-54 (127) 


-162 (220) 


+55 (53) 


MEN: 








Cell Matching Technique 


+328* (175) 


-569* (301) 


-184 (203) 


Linear Control Function Model 


-25 (49) 


-89 (98) 


+145 (147) 



NOTE The level of confidence for each coefficient is indicated by the asterisks following each coefficient (* = 90%; ** = 95%; *** = 
99%). ^ 



In the following chapter, the Heckman selection model is used to derive estimates of average 
JTPA impacts as well as separate activity and termination status impacts for enrollees. The third 
comparison group (Group C), identified in this chapter as being the most similar to JTPA 
enrollees in PY 1987, will be used as a proxy for the JTPA counterfactual. Based upon the 
selection tests run above, the likelihood that tht non-experimental earnings estimates of 
program activity, and termination status impacts are characterized by selection bias are 
minimized. 
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Endnotes 



1. An explanation of the statistical procedures used to control for unobserved differences 
affecting the selection decision and subsequent employment and earnings is beyond the scope 
of this study. The econometric literature often cited to explain this statistical methodology 
includes: Heckman (1979), Bamow, et.al. (1980), Bassi (1983), Heckman and Hotz (1989), and 
Greene (1993). 

2. It may be noted that many unemployed individuals who register with ES do so to be placed 
into full time employment without first having to undergo intensive job training. To the extent 
this characterizes those in the ES comparison group, the non-experimental net impact estimates 
of this study approximate the premium realized by JTPA trainees relative to not undertaking 
any form of government funded training. In contrast, the net impact estimates of the National 
JTPA Experiment (NJE) measure the premium realized by those assigned to JTPA compared to 
the control group who may either register for ES or undertake other forms of training 
self-financed or funded through other government programs. Since not all members of the NJE 
control group registered with a state's employment service, the net impact estimates are likely 
to be smaller than those of the Utah study because more of the NJE control/ comparison group 
may have undertaken non-JTPA II-A funded training. 

3. A random sample was drawn instead of using only those cases coded as disadvantaged 
because Utah Employment Security does not verify the accuracy of this data element on the 
intake form. It is coded according to a self-reporting process whereby the applicant informs the 
Employment Security intake official of his/her status following the viewing of a thirty minute 
movie explaining the eligibility for and services available from the department. 

4. Less than 2% of all individuals were omitted because of missing demographic variables. As 
such, the resulting sample should reflect the general demographic composition of the original 
10% sample of ES registrants. 

5. Observations showing no quarterly wages are composed ot (a) unemployed individuals 
who wish to work but can find no acceptable employment during the quarter, (b) individuals 
who are not working and who are not looking for work (e.g. students, housewives, the 
institutionalized, etc.), and (c) individuals employed but not in Utah's covered sector (e.g. 
self-employed or those employed outside the state). 

6. The "dip" in earnings of JTPA participants is characteristic of the pattern of pre-enrollment 
earnings of those enrolling in most federally funded job training programs. This pattern was 
first analyzed for Manpower Development and Training Administration (MDTA) enrolleesby 
Ashenfelter (1978) and later with Comprehensive Employment and Training Act (CETA) 
enrollees by Ashenfelter and Card (1985). 

7. Three independent studies using the CPS-SSA data for deriving comparison groups to 
analyze the NSW are found in: LaLonde (1986), Fracker and Maynard (1987), and Heckman 
and Hotz (1989). 

8. See for example: Long and Wissoker (1992) and Priedlander and Robins (1992). 

9. As noted ear r, one way an estimate of JTPA training impact is said to be biased is when 
factors which affect the decision to en roll in JTPA are omi tted from the model and are also related 
to the post-training outcome measures of net impact studies. When this occurs, the independent 
JTPA training variable is correlated with the error term in the net impact model - causing 



estimators to be biased. See Heckman and Robb (1985), Heckman and Hotz (1989), and Moffit 
(1992) for a detailed statistical explanation of the selection bias problem. Other biases, such as 
measurement error bias, may also characterize the non-experimental models, but are not 
considered to be significant in this study. 

10. These techniques are described in Rubin (1980) and Rosenbaum and Rubin (1985) and later 
used in a CETA evaluation study for USDOL by Westat, Inc. and later published by Dickinson, 
Johnson and West (1986). 

1 1 . Since less than 20% of JTPA II-A adult participants in Utah are non-white, it is not possible 
to classify racial ethnic minorities into smaller sub-sets. 

12. A fifteen cell matrix composed of five age ranges (22-25, 26-30, 31-35, 36-40, and 41 to 54) 
and three schooling categories (drop out, high school graduate/equivalent, and post-high 
school) was constructed for each of the four gender-race groups of JTPA trainees. This 
distribution was then replicated by random sampling within each cell of the larger sub-set of 
Emplo}Tnent Security applicants. 

13. Since a relatively large percentage of JTPA and Employment Security registrants are out 
of the labor force in a given year, the wage level distribution and the change in wage distribution 
were first separated into those working and those out of the labor force. The JTPA distribution 
of those out of the labor force was replicated with the Employment Security sample, and the 
remaining distribution of workers was categorized into wage level and wage change quartiles 
with the ranges defined by the observed JTPA distributions. Random samples of Employn^nt 
Security registrants in each of these wage level and wage change distribution categories were 
taken when the number of Employment Security registrants alio- ved. The actual number of 
Employment Security registrants was used in those categories when the number of clients was 
too small to derive the expected distribution of the JTPA enrollee group. 

14. The same methodology for replicating the distribution of the level of earnings for the first 
comparison group is followed here in that the JTPA individuals without wages are separated 
from the remaining quartiles of quarterly wage earners. To the extent the reduced sample size 
allows, this distribution is replicated by random sampling the larger Employment Security 
sample within each category. 

15. As defined earlier, a fully attached worker is one with positive quarterly earnings for the 
four quarter period; a partly attached worker has positive quarterly earnings for one to three 
quarters; and one who is out of the labor force has no positive quarterly earnings for the 
four-quarter period. Two successive four quarter periodsare defined prior to the time a member 
of the comparison group utilized the Utah Employment Security. 

1 6. As before, all wage distributions are first categorized according to individuals not working 
or employed, with the latter being divided into quartiles according to the wage level or wage 
change measure of earnings. To the extent the reduced sample size allows, these distributions 
are replicated by random sampling the larger Employment Security sample within each 
category, 

17. In a statistical sense, omitted variables which affect an independent variable (JTPA 
participation) as well as the dependent variable (say post-training earnings) violate the 
assumption of independence between an independent variable and the error term. This results 
in biased estimators, as the expected value of the error term — given the value of the independent 
JTPA participation variable — is not equal to zero. The goal of specifying non-experimental 
models is to find a way to break this dependence by adding omitted variables to the model 
and/or re-specifying the models in ways to break the dependence between independent 



ERIC 



50 



5: 



variables and the error term. The ability to re-specify the non-experimental models to break 
this dependence rests upon the underlying assumptions one must make regarding the selection 
process and the en'or structure of the models. Further discussion on these points may be found 
in Heckman (1979), Heckman and Robb (1985), and Heckman and Hotz (1989). 

It may be noted that in experimental models, selection bias will not arise as individuals are 
ra. idomly assigned to either the treatment or control group. In this case the selection process is 
random and not systematically related ^.o any independent variables, so that the independence 
between the JTPA participation variable and the error term is maintained. 

18. Since much of their analysis was to compare controls in experimental demonstrations to 
selected comparison groups, many tests were designed to analyze post-training, as well as, 
pre-training earnings. Several of these tests are not relevant to the present study since no control 
group from a JTPA experiment in Utah exists, and as such, only pre-enrollment validity tests 
were undertaken. 

19. See: Ashenfelter (1978) and Ashenfelter and Card (1985) for non-experimental training 
impact models which utilize the longitudinal structure of earnings. 

20. It may be noted that tests for selection bias in all cross-section earnings models using the 
earnings for the second year prior to entry failed. The same was true for the year prior to entry 
for males, which experience a greater "earnings dip" in the immediate pre-entry periods than 
do females. 

21. Another variant of the fixed effects model is the random effects model which requires 
more pre-entry earnings observations than existed in this Utah Study. 

22. For a discussion of these complex assumptions, see Heckman (1979). 

23. As noted earlier, the 6-month period refers to the third and fourth pre-entry quarter that 
occurs just prior to the "dip" in earnings of JTPA enrollees. Since 30 regression models v/ere 
used to test for selection bias, only a listing of dependent and independent variables for the five 
tests for each comparison group is given in Appendix C. Complete regression results are 
available through the Commission. 

24. Placement for those enrolled in classroom occupational skills and job search assistance 
programs is straightforward, whereas placement in OJT programs requires the participant to 
be hired permanently following termination of the wage subsidized period of JTPA training. 

25. It may be noted that for a period during PY 1987, eligible individuals were permitted to 
find their own OJT employer - which may well have resulted in a different type of person 
selecting OJT compared to the other activities offered by Utah's SDAs. 

26. It may be noted that the National JTPA Experiment design precludes experimental 
estimates of activity enrollment and termination status impacts as both are subject to individual 
choice — something the random assignment experiment is not designed to address. Failure to 
capture such individual choice seriously limits the understanding of how JTPA works. Such an 
understanding is crucial to improving the effectiveness of future JTPA program operations. It 
is for this reason, that non-experimental impact models need to be developed and improved. 
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■ CHAPTER FIVE: Utah JTPA II-A 
Adult Program Impacts 

Net impacts of the Utah JTPA II-A adult programs for PY 1987 are estimated using 
non-experimental techniques, which compare the observed outcomes of program participants 
with those of the ES comparison group derived in Chapter Four. The post-intervention work 
history of the ES group is used to approximate the "counterfactual" - the employment and 
earnings of program participants had they not enrolled in JTPA.^ To allow for participants to 
complete their training, the second year following entry is used as the appropriate period for 
analyzing two major labor market outcomes - the likelihood that a participant is employed and 
the earnings of those who are employed.^ 

This chapter presents the findings of program impact models for all adults who enrolled for the 
first time in the Title II-A program in Utah during PY 1987. The findings are presented in two 
sections. The first section begins with a discussion of the differences in the observed, or 
statistically "unadjusted" f>ost-entry employment and earnings of enrollees and those of 
members of a similar comparison group. Then these observed differences are contrasted to the 
statistically "adjusted" differences, which are derived from models of post-program 
employment and earnings that adjust for both observed and unobserved characteristics of 
participants.^ 

One purpose of the discussion is to demonstrate that simple bar chart descriptive studies, which 
indicate the differences in unadjusted labor market outcomes between the two groups, may be 
misleading indicators of the program's performance. The second purpose is to demonstrate the 
extent to which JTPA "works," as measured by the estimates of program impacts, and the degree 
JTPA provides assistance to identified "hard-to-serve" clients in the state of Utah. 

In the second section of this chapter, the findings of the statistical models are subjected to 
"sensitivity analyses" for the purpose of determining how much the estimated program impacts 
change if major alterations are made in the construct of the models. This latter analysis is 
important because often only one set of statistical findings is presented whi^^li, in fact, could be 
misleading or simply suspect if the models are poorly or improperly specified. 



PARTI: Unadjusted Veisus Adjusted Program Impacts 



As indicated earlier, the vast majority of JTPA training is composed of relatively short duration 
intervention strategies - most lasting less than 6 months with nearly all being less than 12 
months in duration. Accordingly, program impact measures emphasized in this study are 
calculated during the second year following entry into JTPA, which is clearly in the 
"post-training" period. First, the unadjusted difference between JTPA participants and 
Employment Security (ES) comparison group members (Group C) are compared with the 
"point" estimates of each adjusted program impact. Next, a probability level of statistical 
significance is attached to each "point estimate" to give a sense of a level of confidence that can 
put in the program impact estimates derived from the adjusted models. 
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POINT ESTIMATES OF PROGRAM IMPACTS 



According to the descriptive statistics shown in Figure 5.1 below, the JTPA enrollment group 
realizes mixed post-program work experiences compared with the "similar" ES comparison 
group members. Such descriptive findings, if accepted, could suggest that while some 
enrollees remain employed, the jobs they find pay less than those they would have found on 
their own — based on the experiences of the comparison group. 

The problem with such descriptive information is that these statistically "unadjusted" 
differences between the two groups do not account for either observed or unobserved 
differences that exist between JTPA enrollees and members of a "similar" comparison group. In 
reality, the differences in post-program employment and earnings are affected first by observed 
differences in demographic characteristics (such as age, race, marital, and welfare status among 
others), schooling levels, and pre-entry work histories of the two groups. Secondly, differences 
in the outcomes are affected by differences in unobserved factors including among others, 
differences in attitudes toward self-improvement, motivation to work, and English language 
barriers between the enrollment and comparison groups. These factors, in conjunction with 
enrolling in IT? A, result in the unadjusted differences in post-program work experiences 
depicted in Figure 5.1 . The objective of building statistical models of post-program employment 
and earnings is to separate the JTPA impact from the other observed and unobserved factors 
which are combined in simple bar graph descriptions. 

The significant differences between employment and earnings levels of adult women and men 
seen in Figure 5.1 reinforce the widely understood fact that the labor market processes for the 
two gender groups are very different. For this reason, statistical models are developed 
separately for adult women and men. For each gender, a complete model is derived which first 
accounts for (or in econometric language "controls for") observed differences between JTPA 
enrollees and the ES comparison group (Group C) selected. In addition, unobserved factors 
which affect an individual's choice to enroll in JTPA and later affect post-program employment 
and earnings are included in the models through a complex statistical procedure. The full set 
of results is given in Appendix D; the relevant JTPA program impact results are given below in 
Table 5.1.'' 

Average TTPA Program Impacts. Tlie adjusted program impact is measured as the difference 
between the program outcomes of participants and the outcomes estimated to exist had the 
participants not enrolled in JTPA. Two features shown in Table 5.1 are noteworthy with respect 
to the adjusted impacts compared with the unadjusted outcomes described above. First, the 
adjusted employment rate impact estimates are similar to the unadjusted differences although 
they are somewhat higher (+5.1% versus +4.5% for adult women and +7.5% versus +6.0% for 
adult men). Second, the adjusted earnings impacts estimates differ significantly from the 
unadjusted differences. 

Adult women participants are now estimated to realize $510 higher annual earnings in the 
second post-program year because of program participation even though the mean unadjusted 
difference between participants and comparison group members is a negative $326. The reason 
the figures differ so much is that JTPA participants may have less human capital (i.e., both 
weaker education and work histories) and/or perhaps weaker work motivation or other 
non-measured factors related to later employment and earnings potential. Failure to capture 
these observed and unobserved factors, while only slightly biasing the employment rate 
program impact, seriously biases the earnings impact downward by over $800 in the second 
year following entry. 
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FIGURE 5.1 
Work Experience In Second Year 
Following Entry Date 
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TABLE 5.1 

COMPARISON OF UNADJUSTED WITH ADJUSTED PROGRAM IMPACTS 
Dt TJNG SECOND YEAR FOLLOWING ENTRY* 



(A) ADULT WOMEN 


JIPA PROGRAM 


EMPLOYMENT RATE 


EARNINGS IF EMPLOYED 


CHARACTERJSnC 


UNADJUSTED 


ADJUSTED 


UNADJUSTOD 


ADJUSTED 


JTPA TOTAL 


+ .045 


+ .051 


-326 


+510 


JTPA ACnVITY: 










CLASSROOM TRAINING 


+ .050 


+ .052 


-113 


+ 849 


ON-THE-JOB TRAINING 


+ .096 


+ .101 


-367 


+ 1026 


OTHER ACnVITY 


-.035 


-.040 


^36 


-278 


TERMINATION STATUS: 










PLACED 


+ .081 


+ .087 


+520 


+ 905 


NOT PLACED 


-.069 


-.066 


-2463 


-1771 


NOT COMPLETE 


+ .052 


-.046 


-3342 


-2040 



(A) ADULT MEN 



JTPA PROGRAM 
CHARACTERISTIC 


EMPLOYMENT RATE 


EARNINGS IF EMPLOYED 


UNADJUSTED 


ADJUSTED 


UNADJUSTED 


ADJUSTED 


JTPA TOTAL 


+ .060 


+ .075 


-1786 


+ 444 


JTPA ACnVlTY: 










CLASSROOM TRAINING 


+ .077 


+ .084 


-712 


+ 1103 


ON-THE-JOB TRAINING 


+ .059 


+ .074 


-1674 


+ 3% 


OTHER ACTIVITY 


+ .050 


+ .068 


-2606 


-77 


TERMINATION STATUS: 










PIACED 


+ .089 


+ .104 


-1319 


+ 1303 


NOT PLACED 


-.006 


+ .032 


-5811 


-780 


NOT COMPLETE 


-.143 


-.120 


^736 


-2054 



'Adjusted program impact estimates are derived from two stage Heckman-type regressions using Group "C" 
from Chapter Four as the representative comparison group. See Appendix D, Tables D.l through D.4 for 
complete listings of these regressions. 



Likewise, the program impact for adult men is biased downward when observed and unob- 
served factors related to later employment and earnings are omitted. In this case, adult men are 
expected to realize a $444 gain in the second post-program year because of program participa- 
tion even though the unadjusted difference is nearly $1,800 less. 

These figures suggest while simple comparisons may be good approximations for 
employment rate impacts, they seriously distort the impact JTPA appears to have on the 
earnings of employed participants. In particular, once both observed and unobserved 
differences between program enroUees and similar ES comparison group members are 
accounted for, JTPA has positive effects on both the employment and earnings experiences of 
enrollees in the selected follow-up period. 

In addition to the over-all impact JTPA participation may have on enrollees, it is also useful to 
identify the different impacts various types of training activities have, as well as the impact 
program completion and placement have on later enrollee employmc^nt and earnings. 

Impacts by Types of Training Activities . As indicated in Table 5.1, three types of major training 
activity are examined for each gender group.^ For women, on-the-job training (OJT) is 
associated with the greatest earnings impact (+$1,026); next occupational classroom training 
(OCT, with an estimated earnings impact of +$849); and finally other activities (OA, with an 
estimated earnings impact of -$278 compared with the ES comparison group). For men the order 
of training activity impacts is reversed.'' The greatest earnings impact for men is realized by 
those enrolled in OCT (+$1,103), while positive but smaller earnings impacts are realized by 
men in OJT programs (+$396).® Similar to women, "other activities" has a negative earnings 
impact for adult men (-$77). 

It may be noted that the program impact findings presented above are possible because of the 
availability of work experience data for the selected comparison group. While this information 
may not be easily obtained in other states, it is still possible to obtain "gross impact" estimates 
of alternative types of training activities using only JTPA terminees. In particular, estimates of 
gross impacts may be derived with statistical models of post-program outcomes of those 
enrolled in more intensive OCT and OJT participants compared with those enrolled in less 
intensive OA programs. Such models for Utah are analogous to the ones used in the National 
Conurdssion for Employment Policy training choices and relative outcomes study, which 
analyzed employment and earnings outcomes for the population of JTPA terminees from 11 
states with UI wage records.^ A discussion of such impact models for Utah, and how they relate 
to the net impacts estimated in the study, is given in Appendix E. 

Impacts of Termination Status. Three types of termination status are recorded for JTPA 
participants in Utah: individuals who complete training objectives and are placed into 
unsubsidized private sector employment, those who complete the program but are not placed 
into jobs, and those who fail to complete the program objectives. 

As indicated above in Table 5.1, placement activities of JTPA appear to be extremely important. 
The estimates of the adjusted emplojonent and earnings impacts for those placed into fully 
unsubsidized private sector jobs are significantly greater than the average impact estimates 
discussed earlier.^^ Adult women and men are 9 to 10% more likely to be employed later if they 
were initially placed as con pared with the smaller estimated average JTPA employment impact 
(i.e., 5 to 8 percentage point premium of the average program participant regardless of 
termination status). In addition, among those employed during the second post-program year, 
lx)th women and men who were placed later earned more than JTPA enrollees in general. 
Program completion and placement are associated with earnings premiums substantially above 
those reported earlier for all JTPA enrollees combined; adult women who were placed earned 
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over $900 more and adult men earned over $1^00 more, relative to members of the comparison 
group. 

In general, individuals who complete the program but are not placed fare better than those who 
fail to complete JTPA, but program completion without placement does not carry the same 
positive post-entry employment and earnings effects as does completion with placement. Adult 
men who drop out of JTPA are far less likely to be employed later (-12%) and earn far less 
(-$2,054) than estimated had they not enrolled in JTPA. Those who complete JTPA but are not 
placed also fare less well, but the differences are smaller (i.e., they are slightly more likely to be 
employed later (+3.2%) and earn slightly less ($780)). 

For adult women, the preirdum to job placement is even greater. Those who complete, but fail 
to be placed, experience roughly the same reduced employment and earnings if employed as 
do those who drop out of JTPA before completion. Both types of individuals are roughly 5% to 
6% less likely to be employed later and earn roughly $2,000 less than those in the comparison 
group. These findings suggest JTPA placement activities are extremely important, especially 
for adult women who typically have less prior attachment to the work force than men. 



CONFIDENCE LEVELS OF ADJUSTED PROGRAM IMPACT ESTIMATES 

Before adjusted estimates of program impacts are used for policy making, it is critical to indicate 
the probabilistic nature of the point estimates discussed above. This is necessary because no 
single statistical model can ever include all variables that affect program impacts and no single 
statistical model of individual's employment decisions or earnings processes can be specified 
in a nwnner that fully accords with reality. In all cases, a margin of error must be added to these 
single "point" estimates which indicates the degree of confidence a researcher has in concluding 
the program did indeed have an effect on employment or earnings. As such, the program 
estimates in Table 5.1 are duplicated in Table 5.2 below along with the "confidence" or 
probability level, one can attach to the point estimates. 

Another interpretation of this probabilistic nature of program impact estimates is the likelihood 
that the impact point estimate has either a positive or negative effect. If the probability is low, 
one could equally conclude the impact estimate is zero. In essence, the probability numbers 
included in Table 5.1 indicate the likelihood of the point estimate being "non-zero," that is of 
having a significant positive or negative effect on program impacts. 

Generally, estimates are said to be "statistically significant" if the probability level of the point 
estimate is equal to or greater than 90%. For example, while a 11 program impacts on employment 
for adult women in the second post-program year are positive, only tv/o estimates are 
considered to be "statistically significant": OJT enrollment (+10.1%) and completing and being 
placed into a job at ternrunation (+8.7%). Using the tests of significance, one can conclude that 
adult women enrolled in OJT (or who are placed into a job at termination) will have a higher 
probability of being employed 2 years after enrollment than their counterparts not enrolled in 
OJT (or placed in a job at termination).^^ Enrollment of women in OCT' programs may increase 
the probability of employment, but the confidence one places in this estimate is less. Similarly, 
the confidence of the positive estimates for employment among adult men is significantly higher 
than that of the impacts on earnings. 
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TABLE 5.2 

ADJUSTED PROGRAM IMPACT ESTIMATES AND PROBABILITY 
OF ESTIMATES BEING NON-ZERO 



(A) ADULT WOMEN 




JTPA PROGRAM 


EMPLOYMENl^ RATE 


EARNINGS IF EMPLOYED 


CliARACTERlSnC 


ADJUSTED 


PROB LEVEL 


ADJUSTED 


PROB LEVEL 


JTPA TOTAL 


+ .051 


88% 


+ 510. 


68% 


JTPA ACTIVITY: 










CLASSROOM TRAINING 


+ .052 


85% 


+ 849. 


80% 


ON-THE-JOB TRAINING 


+ .101*** 


97% 


+ 1026. 


88% 


OTHER ACnVITY 


-.040 


72% 


-278. 


29% 


TERMINATION STATUS: 










PLACED 


+ .087*** 


97% 


+ 905. 


84% 


NOT PLACED 


-.066 


68% 


-1771. 


83% 


NOT COMPLETE 


-.046 


75% 


-2040.*** 


97% 


(A) ADULT MEN 




JTPA PROGRAM 
CHARACTERISTIC 


EMPLOYMENT RATE 


EARNINGS IF EMPLOYED 




ADJUSTTED 


PROB LEVEL 


ADJUSTED 


PROB LEVEL 


JTPA TOTAL 


+ .075*** 


99% 


-r444. 


46% 


JTPA ACnVTTY: 












CLASSROOM TRAINING 


+ .084** 


96% 


+ 1103. 


87% 


ON-THE-JOB TRAINING 


+ .074** 


98% 


+ 396. 


40% 


OTHER ACnVTTY 




96% 


-77. 


8% 




TERMINATION STATUS: 










PLACED 




99% 


+ 1303.** 


96% 




NOT PLACED 


+ .032 


38% 


-780. 


36% 




NOT COMPLETE 


-.120*** 


99% 


.2054.** 


97% 





NOTE: The level of probability that the estimated program impact is non-zero (i.e., having no impact) which 
is statistically significant, varies from 90% (minimally acceptable probability) to 99% (highest level of probability). 
The three major probability levels are noted by the number of asterisks appearing after a statistically significant 
estimate: * = 90%; ** = 95%; and *♦* = 99%. 
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A major factor contributing to the relatively low significance of the point estimates reported in 
this study is related to small sample size. Unlike the National JTPA Experiment which sampled 
over 20,000 individuals or even the National Commission for Employment Policy's impact study 
based upon 10,000 adult women and 10,000 adult men from the universe of JTPA terminees in 
eleven states, the Utah study is based on 1,655 adult women and 1,737 adult men. While this 
size is sufficiently large to derive stable point estimates from individual data, they may not be 
stated with the same level of confidence as multi-state or nationally based studies. 

Trade-offs may need to be made when undertaking program impact analyses. On the one hand, 
states may not want to apply "statistically significant" estimates of program impacts from large 
national (or multi-state) JTPA studies to their own state because they believe their state has its 
own distribution of JTPA clientele or particular anomalies regarding its regional and local labor 
markets. On the other hand, state-specific studies - which can be adjusted for these anomalies 
- are smaller in size and often result in lower levels of confidence that the estimates of JTPA 
program impacts are "meaningful" (i.e., are likely to have a strong positive effect). 

In summary, the employment impact estimates for Utah's Title II-A adult enrollees for PY 1987 
suggest JTPA participation has a strong positive effect on employirient for adult women and 
men who complete more intensive training programs and who are placed into fuliy 
unsubsidized private sector jobs. JTPA resources directed towards job placement, in addition 
to job training, appear to be beneficial in Utah. This positive employment impact during the 
second post-program yer.r in Utah for PY 1987 is somewhat greater for adult men than adult 
women. In addition, JTPA results in higher second year earnings, if employed, for those placed 
through more intensive training programs. For adult women, this earnings impact is stronger 
for those enrolled in OJT, while for adult men it is stronger for those enrolled in classroom 
training. 

Finally, not all JTPA terminees from a particular activity benefit equally from program 
participa Hon. Those who drop out may even be worse off if they had never enrolled. The time 
spent in a program and the knowledge of program failure may have a harmful effect on those 
individuals who, otherwise, do not benefit from JTPA participation. 



THE "HARD-TO-SERVE" ADULTS IN UTAH 

While the focus of the study is on estimating JTPA program impacts, the statistical models may 
also be used to identify which of the personal demographic factors are characteristic of 
"hard-to-serve" enrollees. That is, the complete statistical models contain personal demographic 
factors along with JTPA intervention factors which are both related to post-enrollment 
employment and earnings. (The complete model results are given in Appendix D.) Those 
persona] factors that are significantly related to a smaller probability of being employed in the 
second year following enrollment, and/or significantly related to lower earnings if employed, 
are indicative of "hard-to-serve" individuals. 

The first major finding in this vein is that personal demographic factors are not related to 
post-training employment barriers in finding a job, but rather to barriers in realizing hieh er 
earnings when employed. As shown in Table 5,3, two personal demographic variables - high 
school drop-outs and being black - present significant barriers to higher post-enrollment 
earnings for adult women and men. Female high school drop-outs earn $1,437 less than high 
school graduates, while male high school drop-outs earn $1,688 less; black women earn $1,463 
less than white women, while black men earn $3,777 less than white men. In addition, women 
who were former welfare recipients earn less than non-recipients (-$1,355) as did those residing 
in rural relative to urban SDAs (-$1,148). Men who experienced a "dip" in earnings over the 
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TABLE 5.3 



DEMOGRAPHIC CHARACTERISTICS OF THE "HARD-TO-SERVE" 



DEMOGRAPHIC CHARACTERISTIC 


POST-ENROLLMENT 
EARNINGS 


UKELIHCOD OF 
ENROLUNG INJTPA 


WOMEN: 






Hk - HOOL DROP-OUT 


-$1,437. 


Same As Others 


BLACK (Non-hispanic) 


-$1,463. 


Same As Others 


WELFARE RECIPIENT 


-$1,355. 


More Likelv 


RURAL SDA 


-$1,148. 


Less Likelv 


MEN: 






HIGH SC i 'OOL DROP-OUT 


-$1,688. 


Same As Others 


BLACK (Non-hiispanic) 


-$3,777. 


Same As Others 


PRIOR EARNINGS "DIP" 


-$183/$1,000 Decline 


More Likelv 



2-year pre-enroUment period also realized lower post-enrollment earnings than those not 
characterized by pre-entry earnings declines (-$183 per $1,000 decrease). 

A complicating factor in improving the welfare of the "hard-to-serve," is the ability of SDAs to 
recruit and provide services to this sub-set of program eligibles. This phenomenon is shown in 
Table 5.3 by the relationship between the personal demographic factors and the likelihood of 
program enrollment (These relationships were estimated in the first step of the two stage 
statistical program impact models.) SDAs in Utah did not appear to have a difficult time in 
recruiting and enrolling high school drop-outs and blacks. They also were very successful in 
enrolling women welfare recipients and men who had experienced a "dip" in pre-enroUment 
earnings. 

Women in Utah living in rural SDAs relative to those living in urban SDAs, were less likely to 
enroll in Title II-A programs, however. This may have been due to a failure of these SDA^» to 
actively recruit economically disadvantaged women, and/or the eligible women were less 
interested in enrolling in JTPA. Whatever the reason, women in rural SDAs appear to be the 
most "hard-to-serve" in Utah; they are less likely to enroll, and those that do enroll, earn 
substantially less if employed in the post-enrollment period. 



PARTIL Sensitivity Of Program Impact Estimates 



The adjusted net program im . \s discussed above are derived from a particular statistical 
model with a particular comparison group. The model and comparison group were not chosen 
lightly, however. Their selection was based on an analysis of prior finding?- and alternative 
econometric techniques discussed in the economics literature on program evaluations. 

This point is important because often only one set of findings is presented. Policy makers who 
are not fully versed in the literature on employment and training evaluations or in econon^etrics 
are forced to accept results at face value; they must trust the analysis is both complete and 



ERIC 



61 



objective. This section discusses alternative ways that the statistical analysis in this report could 
have been conducted and indicates how the outcomes can differ, and in fact be misleading, 
depending upon the approach taken. While there are many alternative approaches that could 
have been included, the ones selectc-d highlight key aspects of evaluation research: the choice 
of comparison groups, the availability of data on pre-program employment and earnings 
experiences, and adjustments for unobserved selection bias (through self-selection or selection 
by program operators). 



THE IMPORTANCE OF A WELL CHOSEN COMPARISON GROUP 

A major choice common to non-experimental impact models is that of identifying a comparison 
group similar to JTPA participants. To indicate the importance of choosing the best available 
comparison group, the same specified statistical models used earlier are re-estimated with the 
first of the three potential comparison groups (i.e.. Group A) identified in Chapter Four. The 
key difference between this comparison group and the one chosen for analysis in this study (i.e.. 
Group C) is the emphasis on matching the distribution of p>ersonal demographics rather than 
prior earnings patterns over the 2-year period prior to enrollment. As noted in Chapter Four, 
while Group C minimized selection bias. Group A resulted in the greatest selection bias for both 
adult women and men. 

Table 5,4 shows the results associated with using the comparison group having the greatest 
self-selection bias (identified as "dissimilar") and repeats the results given earlier based upon 
the comparison group with the least selection bias (identified as "similar"). While the estimates 
of the emplo)Tnent rate impacts seem to differ little between using either comparison group, 
the estimates of earnings impacts differ widely. Use of the "dissimilar" comparison group 
seriously biases the estimates of adult women post-program earnings; indeed these estimates 
differ from those derived with the superior comparison group (Group C) by over $2,000. The 
estimates of earnings impacts of adult men differ from that using the sup)erior comparison group 
by over $1,000, Such different estimates of program impacts clearly show the importance of 
identifying a comparison group that is similar to the JTPA participant sample.^^ 



IMPROPER SPECIFICATION OF PRE-ENTRY WORK EXPERIENCE 

A modeling specification that weakens non-experimental studies of program impacts occurs 
when data bases do not contain significant historical employment and earnings information, for 
example, when UI earnings record data are not available. If only that experience just prior to 
entry is used in program impact models, misinformation is likely to arise for those enroUees 
having atypical or temporary work patterns. Many adult JTPA enrollees are individuals who 
only decide to enter the program after going through difficult times in the labor force. Their 
pre-entry employment and earnings records are poor indicators of the stock of human capital 
they have acquired over longer periods of work experience. 

To demonstrate the importance of having more than one year's worth of information on a 
person's work history, this alternative uses only the work experience occurring in the year just 
prior to enrollment for both the JTPA enrollees and the ES comparison group members. The 
results of this specification are listed under the column labeled "one year" in Table 5.5. All the 
information in the second year is purposely omitted, which prohibits the common "dip" in 
earnings from being included in the impact models, especially for adult men. (The colunms 
labeled "two years" refer to the full model version.) 
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TABLE 5.4 



SENSITIVITY OF PROGRAM IMPACT ESTIMATES 
TO SIMILARITY OF COMPARISON GROUPS 



(A) ADULT WOMEN 





EMPLOYMENT RATE 


EARNINGS IF EMPLOYED 


CHARACTERISTIC 


DISSIMILAR 


SIMILAR 


DISSIMILAR 


SIMILAR 


JTPA TOTAL 


+ .036 


+ .051 


+ 2937.*** 


+ 510. 


JTPA ACTIVITY: 










CLASSROOM TRAINING 


+ .038 


+ .052 


+3519.*** 


+ 849. 


ON-THE-JOB TRAINING 


+ .0i-9** 


+.ior* 


+ 3711.*''* 


+ 1026. 


OTHER ACnVTTY 


-.056 


-.040 


+ 251L*** 


-278. 


TERMINATION STATUS: 










PlJ\CED 


+ .067** 


+ .081** 


+3775* 


+ 905. 


NOT PLACED 


-.086 


-.066 


+ 1413. 


-1771. 


NOT COMPLETE 


-.062 


-.046 


+ 1237. 


-2040.* * 


(B) ADULT MEN 




JTPA PROGRAM 


EMPLOYMENT RATE 


EARNINGS IF EMPLOYED 


CHARACTERISTIC 


DISSIMIL\R 


SIMILAR 


DISSIMILAR 


SIMILAR 


JTPA TOTAL 


+ .069*** 


+ .075* •• 


-2688.* •• 


+ 444. 


JTPA ACnVTTY: 










CLASSROOM TRAINING 


+.064** 


+ .084** 


-2823.*** 


+ 1103. 


ON-THE-JOB TRAINING 


+.079'' 


+ .074** 


-1590/ • 


+ 396. 


OTHER Acnvmr 


+.067** 


+ .068** 


-2169.* •• 


-77. 


TERMINATION STATUS: 










PLACED 


+.098*'* 


+ .104*** 


-1369.* • 


+ 1303.** 


NOT PU^CED 


+ .037 


+ .032 


-3609.* 


-780. 


NOT COMPLEl-E 


-.125*** 


-.120* • 


-4767.* 


-2054.* * 



NOTE: See note on Table 5.2 for an explanation of the three probability levels indicated by the 
asterisks on relevant program impact estimates. 
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As shown in Table 5.5, omission of the second pre-entry year in the "truncated model" has little 
effect on the estimates of employment rate impacts. However, again there are sizable differences 
between the two models in their estimates of earnings impacts. The tnmcated model produces 
significantly larger estimates of earnings impacts for adult women and men. For example, 
omission of employment and earnings information in the second year prior to entry results in 
an estimated $2,000 earnings impact for women and a $5,000 earnings impact for men. These 
estimates are considerably greater than reported earlier in models which specify an additional 
pre-entry year (i.e., $510 earnings impact for women and $444 earnings impact for men). The 
higher earnings impacts occur because the low pre-entry earnings are assumed to reflect a 
permanently low stock of human capital in the truncated model. Once the higher levels of earlier 
pre-entry earnings are included in the full model construct, lower earnings impacts are esti- 
mated. 



OBSERVED VERSUS UNOBSERVED ADJUSTMENTS 

The final characteristic of the models used in this study is the specification of both observed and 
unobserved differences between the JTPA participants and ES comparison group members. To 
determine the importance of controlling for selection bias in non-experimental model designs, 
an alternative model is constructed that accounts only for the observed differences between the 
two groups. (In Table 5.6 the columns labeled "observed only" refer to this alternative.) Results 
of this model are compared to those of the adjusted model used throughout this study, which 
account for both observed and unobserved factors in a two stage statistical analysis. (The 
columns in Table 5.6 labeled "observed and unobserved" refer to the full model version.) As 
shown in Table 5.6, estimates of the "observed only" model version are fairly good 
approximations of the program impacts on employment rates. But, like before, they are poor 
approximations of the program impacts on earnings of those employed. For example, the 
program impact estimate of JTPA enrollment is nearly doubled for the single stage models. The 
program impact estimate for adult women is $805 and for men is $934 in the second 
post-program period; both are significantly greater than estimated with the two step model 
reported earlier. 

These results suggest individuals who are more likely to self-select into JTPA (or be selected for 
JTPA) are also more likely to find better paying jobs in the post-training period compared to 
persons in the ES comparison group. Thus, with a single step model, part of the increase in 
post-entry earnings is due to program participation and part is due to program participants 
whose unobserved characteristics are positively related to higher earnings. As discussed earlier, 
these unobserved factors may include greater work motivation, greater self-esteem, or fewer 
English language problems. 
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TABLE 5.5 

SENSITIVITY OF PROGRAM IMPACT ESTIMATES TO 
SPECIFICATION OF PRE-ENTRY WORK EXPERIENCE 



(A) ADULT WOMEN 



JTPA PROGRAM 
CHARACTERISTIC 


EMPLOYMENT RATO 


EARNINGS IF EMPLOYED 


PRE-ENTRY WORK EXPERIENCE: 


PRE-ENTTRY WORK EXPERIENCE: 


ONE YEAR 


TWO YEARS 


ONE YEAR 


TWO YEARS 


JTPA TOTAL 


+.056* 


+ .051 


+ 1921."' 


+510. 


JTPAACnVTTY: 










cij\ssroom training 


+ .054 


+ .052 


+ 1892."' 


+849. 


ON-THE-JOB TRAINING 






+ 22A9.*" 


+ 1026. 


OTHER ACnVITY 


-.038 


-.040 


+850. 


-278. 


TERMINATION STATUS: 










PIj\CED 




+ .081*' 


+ 1634.»" 


+905. 


NOT PL\CED 


-.053 


-.066 


-675. 


-1771. 


NOT COMPLETE 


-.044 


-.046 


-1013. 


-2040." 



(B) ADULT MEN 



JTPA PROGRAM 
CHARACTERISTIC 


EMPLOYMENT RATE 


EARNINGS IF EMPLOYED 


PRE-ENTRY WORK EXPERIENCE: 


PRE-ENTRY WORK EXPERIENCE: 


ONE YEAR 


TWO YEARS 


ONE YEAR 


TWO YEARS 


JTPA TOTAL 


+ .069"' 


+ .075*- 


+5004."* 


+444. 


JTPA ACnVTTY: 










CLASSROOM TRAINING 


+ .064""" 


+ .084" 


+4539."- 


+ 1103. 


ON-THE-JOB TRAINING 


+ .086*" 


+.074" 


+ 3881."* 


+396. 


OTHER ACnvITY 


+ .028 


+.068" 


+3380."' 


-77. 


TERMINATION STATUS: 










PL\CED 


+ .097"* 


+ .104"' 


+5178."* 


+ 1303." 


NOT PLACED 


-.014 


+ .032 


+5186."* 


-780. 


NOT COMPLETE 


-.069* •• 


-.120* " 


+ 3786."* 


-2054." 



NOTE: See note on Table 5.2 for an explanation of the three probability levels indicated 
by the asterisks on relevant program impact estimates. 
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TABLE 5.6 



SENSITIVITY OF PROGRAM IMPACT ESTIMATES 
TO SELF-SELECTION BIAS CORRECTION 

(A) ADULT WOMFN 



JITA 

PROGRAM CHARACrERJSllC 


EMPLOYMENT RATE 


EARNINGS IF EMPLOYED 


ADJUST MODEL TO ACCOUr^ 
FOR: 


ADJUST MODEL TO ACCOUNT FOR: 


OBSERVED 
ONLY 


OBSERVED AND 
UNOBSERVED 


OBSERVED 
OKLY 


OBSERVED AND 
UNOBSERVED 


JlrA iOlAL 


+ .052*' 


+ .051 


+ 805.* 


+ 510. 


JITA ACnVITY: 










CLASSROOM TRAINING 


+ ,052* 


+ .052 


+ 1021.* 


+ 849. 


ON-THE-JOB TRAINING 




+.101** 


+ 1420.*** 


+ 1026. 


OllIER Acnvn'Y 


-.041 


-.040 


-747. 


-278. 


TERMINAllON STATUS: 










PLACED 


+.082* 


+.081** 


+ 1745.*** 


+ 905. 


NOT PLACED 


-.066 


-.066 


-2580.* • 


-1771. 


NOT COMPLETE 


-.045 


-.046 


-2607.**' 


-2040. 



(A) ADULT MEN 



JTPA PROGRAM 
CHARACTERISTIC 


EMPLOYMENT RATE 


EARNINGS IF EMPLOYED 


ADJUST MODEL TO ACCOUNT 
FOR: 


ADJUST MODEL TO ACCOUNT FOR: 


OBSERVED 
ONLY 


OBSERVED AND 
UNOBSERVED 


OBSERVED 
ONLY 


OBSERVED AND 
UNOBSERVED 


JTPA TOTAL 


+ .078*** 


+ .075*** 


+ 934.* 


+444. 


JTPA ACnVITY: 










CI^SSROOM TRAINING 


+ .087* • 


+ .084** 


+ 1542.* 


+ 1103. 


ON-lHE-iOB TRAINING 


+.074*** 


+ .074** 


+ 981. 


+ 396. 


OTHER ACnviTY 


+ .077* • 


+.068** 


+412. 


-77. 


TERMINATION SI A'lTJS: 










PIJ\CED 


+ .108*** 


+.104*** 


+ 1720.*** 


+ 1303.** 


NOT PLACED 


+ .042 


+ .032 


-1669. 


-780. 


NOT COMPLETE 


-.n9** 


-.120* • 


-4330.** 


-2054.* • 



NOTE: Sec note on Table 5.2 for an explanation of the three probability levels indicated by the asterisks 
on relevant program impact estimates. 
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Endnotes 



1 . The use of the entry date, rather than the exit date, as the departure point of analysis is in 
contrast with traditional JTPA follow-up reports. The former should be used in net impact 
analysis because a program eligible individual who enrolls in JTPA would have remained in 
the labor force either working or continuing to search for work beyond the entry date. Reports 
or gross impact studies that focus solely on JTPA enrollees should use the exit date as the 
relevant departure point for describing and analyzing different employment and earnings 
patterns of enrollees. As such, the term "post-program" as used in this net impact study refers 
to the entry not exit date. 

2. Because longer periods of post-entry work histories were not available in this study, it was 
not possible to undertake a full-scale cost-benefit study, which would compare the participants' 
foregone earnings and program costs with the JTPA enrollees' earnings gains, and resulting 
higher tax revenues and reduced welfare expenditures related to their improved work histories. 

3. Selection into JTPA is modeled in the first step. The results are then used in the 
post-program models of the second step. In essence, the unobserved factors which are related 
to program participariou arc included in an error term of the first stage model. This error term 
is then used as an additional explanatory factor in the second step, which explains post-program 
employment and earnings of program participants. 

4. The employment and earnings outcome measures shown in Figure 5.1 are calrulated as: (1) 
the percent of treatment or comparison group members who had positive earnings during the 
second year following entry to JTPA or ES, and (2) the average annual earnings of those who 
had earnings during this period. 

5. The modeling of observed factors has traditionally been carried out with linear or non-linear 
multivariate regression models, whereby the variation in a labor market outcome, say earnings 
if employed, is explained by a host of causal factors. These models are common to all modem 
statistical software packages. The statistical models utilizing unobserved factors contained in 
the error structures of the two processes of participation and later work history outcomes is 
relatively new and supported in specialized econometric software packages only. This 
procedure uses information about the structure of modeling errors in the program participation 
model along with the post-program work history models. For a statisrical explanarion of this 
sophisticated technique, see: William Greene (1993). The same author has also compiled 
specific computer software to handle such complexities, and is available under the trademark 
"LIMDEP" (Limited Dependent Variables). 

6. In reality, JTPA participants receive minimal supportive services, career counseling, and 
brief short job search assistance in addition to their major activity. In this sense, "activity" as 
used here refers to the major enrollment activity of program enrollees. 

7. Job Search Assistance is the predominant activity in "other activities" included in this study. 

8. Given the self-selection bias found for adult men in classroom training — as distinct from 
other activities — the conclusion that the greatest earnings impact is found for males in OCT 
must be qualified somewhat. This apparent earnings premium is related to the type of 
individual who selects into this activity and who later realizes higher earnings. As explained 
in Chapter Fcur, the statistical models used in this study do not account for selection bias across 
activity types, as none was found to exist for adult women or adult men enrolled in OJT or OA 
programs. 
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9. See Geraci and Romero, forthcoming. 
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10. It may be noted that these estimates have been corrected for selection bias as explained in 
the previous chapter. 

11. A USDOL-OIG report (1988) took the U.S. Department of Labor to task for emphasizing 
placement activities at the expense of training activities for the economically disadvantaged. 
The findings in this study suggest JTPA placement-related expenditures appear to be justified 
for Utah. 

12. Consider the hypothetical situation, where a researcher derives 100 samples of JTPA 
participants from the population. Adult women enrolled in OJT are 10.1% more likely to be 
employed than others in 97 out of the 100 randomly chosen samples. The higher the probability, 
the greater the confidence in the estimate. 

13. It may be noted that another comparison group which was unavailable to the study at the 
time may be composed of individuals who are even more similar than those selected for the 
identified "similar" group. This sub-set of Employment Security participants are those who 
report themselves as "economically disadvantaged" on the ES intake form. It may be of interest 
to determine if this sub-set of individuals who is more "similar" to JTPA participants, especially 
in terms of unobserved factors, results in different program impact estimates. 
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■ CHAPTER SIX: Conclusions 



Since the Job Training Partnership Act OTP A) was enacted in 1982, states have been limited in 
their ability tc evaluate the training programs in terms of improving the post-program 
employment and earnings of participants. States have largely relied upon JTPA's system of 
performance standards that provides descriptive data on program outcomes and only measures 
the relative effectiveness of the programs across local Service Delivery Areas (SDAs). Rigorous 
state-level evaluations of the effectiveness of JTPA as a whole in improving the participants' 
economic welfare have not been undertaken, with a few exceptions.^ 

Two types of evaluations are possible. One is experimental, where ti^eatment and control groups 
are randomly selected from among individuals who seek to participate in JTPA programs and 
are eligible for them. The former receive JTPA services and the latter do not. The employment 
and earnings of these two groups are later compared to determine if the participants of the 
prograiTO are more likely to be employed, and if they are employed, whether their earnings are 
higher. The U.S. Department of Labor (USDOL) undertook such an experiment of JTPA in the 
mid-1980s. 

The second tj'pe of evaluation is non-experimental. Instead of having treatment and control 
groups, this approach uses treatment and comparison groups. The treatment group is 
comprised of JTPA participants, while the comparison group is selected from a group of 
non-participants such as Employment Service registrants. With this technique it is critical that 
both observed and unobserved characteristics of the comparison group are as similar as possible 
to those of the JTPA participants. If this is not the case, then the program impacts that are 
estimated could be due to differences between the two groups' characteristics rather than to the 
program itself (formally termed "selection bias"). 

States have not undertaken non-experimental evaluations for several reasons. One is that 
results from this approach, found in the late 1970s for the impacc of training programs under 
the Comprehensive Employment and Training Act (CETA), were not considered reliable due 
to selection biases. 

Second, in an attempt to overcome these problems, the USDOL undertook an evaluation of JTPA 
using the experimental approach. States have been awaiting the results of the National JTPA 
Experiment rather than undertaking either their own experiments (which are costly) or their 
own non-experimental evaluations (which have not been considered to produce reliable results). 
The problem is that the National JTPA Experiment has not proven to be a panacea for the 
methodological problems involved in evaluating JTPA, as this report has indicated. 

Third, a study demonstrating the feasibility of using non-experimental techniques has been 
lacking. This report helps fill this gap. It uses statistical techniques and data (data on JTPA 
participants and on Employment Security registrants linked with their individual 
Unemployment Insurance (UI) earnings records) that were not available when the earlier 
non-experimental evaluations of CETA programs were undertaken. Also, advances in 
statistical evaluation techniques have increased the confidence that may be placed in 
non-experimental results. Advances in the availability of data and its maintenance over the 
past decade or so now make it possible for states to apply the non-experimental techniques to 
assess their programs. 
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As a cautionary note, and as this study has demonstrated, the results produced by this type of 
analysis are highly sensitive to the comparison group chosen, the specification of pre-en try work 
experience, and the technique used to control for selection bias. Clearly, those respcnsible for 
generating statistical estimates of program impacts should provide policy makers with the 
relevant probability levels associated with point estimates. Care should be taken when choosing 
"similar" comparison groups and when specifying a formulation of statistical program impact 
models. Finally, it may be desirable to develop a few, clearly described, alternative analyses 
and associated impact estimates. 

In this context, it may be useful for individual states to compare the results of their evaluations 
with those of other states. While differences will certainly appear - due to differences in their 
labor markets and particular groups served, for example - the compat isons would reveal areas 
where improvements may be necessary. With these points in mind, states that wish to assess 
their JTPA program, and yet lack the resources to conduct experimental studies, can conduct 
non-experimental studies to determine the likely range of estimates of JTPA program impacts. 

Given the restrictions of sample size in many states, it may prove worthwhile for states to pool 
their UI earnings records so that greater statistical precision can be attached to estimated 
program impacts. In addition, contiguous states might also pool their UI earnings records to 
capture out-of-state employment of residents. 



Major Findings 



Perhaps the most impK)rtant finding from this analysis has been noted above: It is possible to 
develop statistically reliable program evaluations for JTPA through the use of 
non-experimental techniques that rely on earnings records readily available to states. Given 
the emphasis on long-term impacts for JTPA participants - and the benefits that would be 
expected to accrue from a less costly methodology than random sampling, with all its attendant 
application difficulties - the Utah study should be good news for evaluators of JTPA. 

Equ -lly important is the finding that in Utah, the specific state in which these analysis techniques 
were applied, participants who completed and were placed in unsubsidized jobs (or were 
retained by their employer in the case of on-the-job training) had a significantly higher 
likelihood of being employed 2 years after their original enrollment. Both adult women and 
mei' were roughly 10% more likely to be employed. 

Regardless of whether the men had been in classroom occupational training, on-the-job training, 
or "other services" (primarily job search assistance), there was a significantly higher probability 
that they would be employed 2 years after enrollment than their counterparts who had 
registered with the Employment Security but had not participated in a JTPA program. For 
women, however, only on-the-job training was associated with greater employment 2 years after 
enrollment. 

In addition, and especially for men placed into unsubsidized employment, their earnings, 
were significantly higher — nearly $1,000 for women and over $1,300 for men. 

Both of these findings are important for the future of JTPA, since they point to positive long-term 
program outcomes in the one case and to a cost-effective means of tracking these outcomes in 
the other. 
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Other programmatic findings were also observable. In Utah, at least, JTPA programs have 
generally been enrolling individuals whose characteristics are assumed to make them 
"hard-to-scrve." For example, among adult women, welfare recipients are more likely than 
non-recipients to enroll. Similarly, among both adult men and women, school drop-outs are as 
likely as non-drop-outs to enroll. Given the mandate for serving more school drop-outs in the 
JTPA Reform Amendments, however, the ratio of drop-outs to non-drop-outs among 
participants will need to increase in the future. 

The results from Utah on employment outcomes are generally consistent with those of other 
studies. For example, in the National JTPA Experiment, participation in JTPA generally was 
associated with higher rates of employment 18 months after leaving the program.^ 

Results from an analysis of earnings data yielded somewhat more equivocal results. Although 
participants showed higher post-program earnings than members of the comparison group, the 
results on average were not statistically significant at the conventional levels. Within these 
parameters, however, it can be said that on-the-job training (OJT), and to a somewhat lesser 
degree occupational classroom training (OCT), appear to ra^se the earnings of women, if thev 
arc employed . 

Tliese results on earnings are generally consistent with those of other studies. In particular, 
on-the-job training appears to have the greatest post-program payoff. Although the results arc 
not strong in the case of Utah, they were quite strong in the National JTPA Experiment.^^ 

For men, the Utah results suggest that classroom training may raise their earnings, but again 
only if they are employed . These results on earnings are not consistent with other findings. In 
particular, in Utah, men who enrolled in occupational classroom training appeared to have 
benefitted more than those who enrolled in on-the-job training. In the National JTPA 
Experiment, the men who were assigned to on-the-job training seemed to derive the most 
post-program benefits from this activit}'. By comparison, the men who were assigned to 
occupational classroom training did not experience gains in post-program earnings compared 
to their randomly assigned control group counterparts. These inconsistent results for Utah men 
could be associated with the relatively high unemployment that existed during the period under 
examination in industries where men are over-represented (e.g., construction and 
manufacturing). The fact that there is a difference between a state-specific study and a study of 
sites across the states indicates the importance of assessing JTPA programs within indi\-idual 
states. 



Poiiq^ Implications 



Perhaps the most important policy implication from this study can be derived from the 
Conference Report on the lob Traininp Reform Amendments of 1992 , which acknowledged the 
work of the National Commission for Employment Policy in demonstrating "the potential cost 
savings of using the unemployment insurance wage records" and requiring the Secretarx' of 
Labor "to report to Congress on the feasibility of establishing such a database" (p. 140). The use 
of Ul wage records from the State of Utah for this case study of JTPA provides evaluators with 
the kind of information they need to further refine this evaluation methodolog}'. Addilicmal 
studies, using Ul wage records, should be welcomed to build upon this base. 

A second broad policy implication stems from the binding that men and women in general 
appear to benefit from program participation, with both a greater likelih(Kxi of employment 
and higher reported earnings. Given these positive results, one obvious implication is the need 
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for administrators at all levels to work diligently to increase the efficiency of JTPA management 
and the services available to participants. The Reform Amendments of JTPA address these 
issues in many ways, with much greater emphasis than in the past on administrative controls 
to be applied at the local level. This report has demonstrated that it is feasible for states to 
undertake evaluations of their own programs, which can then serve as a basis for program 
improvements. 

Third, with regard to program services - and in keeping with the finding that on-the-job training 
has positive results for both men and women (although the Utah study's findings were not as 
strong as those from the National JTPA Experiment) - efforts should be made to ensure that 
on-the-job training contracts written with private employers continue to be an important JTPA 
activity. Once the concern that these contracts reflect the skill level of the job is addressed {as 
was done in the 1992 JTPA Reform Amendments), o»vthe-job training could be even more 
effective than previously. Similarly, classroom training needs to be focused on jobs for which 
there is a strong demand. 
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Endnotes 



1. See: Reed (1986), Hanna (1988), and Bowman (1988). 

2. See Exhibit S.l, Bloon\ et al., (1991), p. XXXVl. The report did not provide findings on 
en\ployn\ent outcomes for the separate training activities. In addition, neither the Nevada nor 
the Indiana state impact studies derived employment outcomes for their JTPA evaluation. 

3. See Exhibit S.6 in Bloom et al., (1992), p. XLV. As discussed in the text, neither the Nevada 
nor Indiana studies derived separate estimates of impacts by activity type. 
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■ APPENDIX A: Experimental 
Evaluations Of Economically 
Disadvantaged Adult Programs 



As noted in the text, experimental studies of employment and training programs for 
economically disadvantaged adults have often been characterized by design, implementation, 
and foliovv-up complexities, which can introduce statistical bias into estimated program 
impacts. Unlike laboratory experiments in a tightly controlled environm.ent, field experiments 
must be operated in the "human environment" characterized by numerous agents acting in 
different socio-economic and political climates. Many of these differences cannot be observed 
and controlled in experimentation, so rigid implementation designs become necessary to ensure 
the differences occur randomly, rather than systematically, across the treatment and control 
groups. Two large national experiments that targeted economically disadvantaged adults (in 
addition to out-of-school youth) are described below to indicate how even random assignment 
experiments may not yield a unique "true" program impact. 



The National Supported Work Demonstration 



The National Supported Work (NSW) Demonstration targeted adult women who were 
long-term AFEXI recipients, had no children under age 6, and had little if any previous work 
experience. The demonstration was carried out in 10 sites from 1976 second quarter through 
1977 second quarter. The target group of 1,600 adult women was randomly assigned in a 
one-to-one ratio to either a treatment or control group. The treatment group had to work in 
ei :her a private or public sector job with close monitoring by the demonstration staff. The control 
group was precluded from receiving this additional service, leaving them in their same position 
as before: to find employment on their own if they so chose. 

The simple design of this experiment made it easy to maintain the "internal validity" of the 
demonstration. Only one service v^as evaluated and only two target groups were identified. 
The control group was not denied any service for which they otherwise would be eligible. All 
treatment members participated and the assignment to each group (treatment and control) was 
purely random. Problems with this experiment came in the evaluation, rather than in the 
implementation-operation, phase of the demonstration. 

Whereas baseline data were filed for all experimentals, follow-up interview survey data were 
only filed for those individuals for whom they could (1) locate, (2) contact, and (3) obtain a 
response. Due to resource constrainfs of the dem.onstration, the four 9-month periodic 
follow-up interviews for the AFDC target group were halted after the first survey. A single 
contact was made in the fall of 1979, occurring from 27 to 44 months after the baseline interview 
date. By this time 1 ,185 of the initial experimental group (roughly 75% of the sample) completed 
the survey, with others omitted for one of the three reasons listed above. As such, the potential 
for non-response bias came to characterize the program impact estimate. No effort was made 
to obtain post-baseline earnings for all experimentals from administrative sources, such as Social 
Security Administration or Internal Revenue Service data files, to validate the accuracy of recall 
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or to test for non-response bias in the 1979 follow-up. As reported in a later independent study 
by ULonde (1986), the final program impact based upon this 1979 survey was estin^ated " o 
$861 (in 1982 dollars). 

It is also interesting to note that later research using different data cohorts resulted in differing 
experimental program impact estimates for the NSW study. In 1987, Fraker and Maynard 
derived Social Security Administration (SSA) earnings data for all experimentals and 
re-estimated the NSW program impact. This time the experimental program impact was 
reported to be only $351, or $467 in 1982 dollars. Still later, Heckman and Hotz (1989) 
re-analyzed the data and reported a $267 ($357 in 1982 dollars) experimental program impact. 
Apparently, the interview data in the original study produced results of a^program net impact 
roughly twice the program impact later reported with SSA earnings data. 



The National JTPA Experimenr 



The National JTPA Experiment (NJE), begun in 1987, is truly a heroic effort to obtain estimates 
of program impacts.^ For the first time, a random assignment process was used at sites across 
the nation in an attempt to estimate overall program impacts, as well as those of separate 
activities, for economically disadvantaged adult women and men (and out-of-school youth). 
Whereas NSW was a demonstration involving a new program activity, Nj£ was a field-based 
experiment designed to evaluate JTPA as it naturally operates. 

18-Month NTE Evaluation Outcomes , Currently, the first of two post-program evaluations of 
the NJE sites has been completed. A sub-set of the full experimental sample was examined, 
based on results from an 18-month follow-up survey used to estimate overall program earnings 
and employment impacts and separate impacts for major program activities. 

The 18-month NJE evaluation of economically disadvantaged adults finds positive and strong 
employment impacts for JTPA as a whole. Adult men enrollees are 4.8% more likely to be 
employed sometime during the 18-month follow-up period than those in the control group, 
while adult women are 3.5% more likely to be employed. While both adult women and men 
realize a positive earnings impact, only the impact for women is statistically significant. Women 
from the treatment group who enrolled in JTPA are estimated to realize a $873 premium over 
the 6-quarter post-intervention period (or $582 per year) compared with otherwise similar 
women assigned to the control group; adult men from the treatment group who enrolled in 
JTPA experience a slightly higher premium (relative to those assigned to the control group), 
although the earnings differential is not statistically significant. 

In addition, based upon those members of the treatment group originally assigned to one of 
three major activities, only adults assigned to on-the-job (OJT) training experienced earnings 
impacts that were statistically significant. Specifically, adult women in the treatment group 
assigned to OJT were estimated to realize $742 greater earnings over the 18-month period (or 
$454 pei /ear), while adult men assigned to OJT in the treatment group were estimated to^realize 
a $781 (or $521 per year) premium over otherwise similar males in the control group. Other 
activity impact estimates for classroom training and a "catch-all" services group are positive but 
not statistically significant. 

As discussed below, these estimates from the first follow-up survey evaluation are derived in a 
complex programmatic environment, where at times individual choice, rather than random 
assignment, came to characterize referral, enrollment, and placement decisions on the part of 
experimental participants and participating SDAs. 
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The complexities of super-imposing a randomized social experiment onto on-going 
employment and training programs appear to have introduced biases into estimated program 
impact estimates. Further, the estimated program impacts from the first follow-up survey are 
conservative estimates of program performance. Differing types of problems surfaced during 
the eariy, intermediate, and final phases of the NJE due to these complexities, including: (1) site 
selection and design, (2) participant recruitment and activity assignment, and (3) follow-up 
procedures used with experimental participants. 

Early Phase; Site Selection and Design . Initially the experimental design called for Service 
Delivery Areas (SDAs) to be selected according to probabilistic models that would ensure the 
national, representative nature of participating SDAs. Unfortunately, the vast majority of sites 
initially identified chose not to participate.^ In the first place, since knowledge gained from the 
evaluation study was considered a "public good," SDAs could learn about the results without 
having to undergo the difficulties of participation. The marketing of an experimental program 
based upon altruism fell on the deaf ears of too many Private Industry Councils (PICs), locally 
elected officials (LEOs), program administrators, service providers, community representatives, 
and state officials required to reach a consensus for voluntary SDA participation. Second and 
more importantly, the financial compensation offered to SDAs was related exclusively to the 
anticipated higher administration costs and hardly matched the expected implicit costs of 
participating in a rariom assignment evaluation program, namely: (1) ethical costs, (2) 
recruitment costs, (3) activity-assignment costs, and (4) performance-standards costs.'' 

On the first point, many operators were opposed to the ethics of assigning eligible persons to 
JTPA activities on a random basis instead of a staff-determined need basis. This initial reluctance 
was lessened somewhat once it was understood that JTPA is not an entitlement program in 
which all eligibles are guaranteed government assistance. Rather, it is a "slot-limited" program 
where available funds are spent on a first-come-first-served basis. 

In weak labor markets where a strong demand for JTPA can exist, a staff-determined rationing 
device may be no more or less "ethical" than rationing through random selection, since all those 
eligible have equal chances of participation. In fact, owing to "who one knows" and the "old boy 
network," one could argue quite persuasively that random assignment is at least as fair as the 
first-come-first-served system common to JTPA. 

The real ethical problem, however, occurs if SDAs are operating in a strong labor market, where 
jobs are relatively plentiful and in an effort to allocate all program year funding, eligible 
individuals must be actively recruited by the SDA.^ Once an interested and eligible individual 
is recruited, the situation becomes one of leaving actual service receipt to chance rather than 
being based upon need. SDA staff must now pass along judgements (often face-to-face with a 
perspective client) which in essence could deny services to those who would have undertaken 
training in absence of the experiment.^ At first, the JTPA withholding period was set at 30 
months, but was soon reduced to 1 .5 years. Even this 18-month period seemed like an eternity 
to operators who often lose perspective clients if they are not assigned immediately to an 
available program.^ 

A second concern was that the requirement of a control group could exacerbate recruitment and 
outreach problems associated with many SDAs." At the outset, the experimental design called 
for equal numbers of treatment and control group members, but this was soon changed to a 
two-to-one ratio to reduce the required size of the control group ~ including many who 
otherwise would have enrolled in JTPA.^^ Title II-A funding dollars are at times not easily 
allocated to the economically disadvantaged, niost of whom never contact JTPA for assistance.^ 
It is not uncommon for SDAs to have to increase recruitment efforts substantially during the 
last 1 or 2 quarters of the program year to meet enrollment goals, especially in labor markets 
with low unemployment rates. It just so happened that during the period of the NJE, 
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unemployment rates were falling, often quite significantly. The necessity of recruiting a larger 
intake pool to account for those assigned to the control group exacerbated this worsening 
situation. 

Third, the rindom assignment process added a degree of inflexibility as to how participants are 
actually assigned to activities within many SDAs. In particular, the design called for the random 
assignment of individuals after a service activity was chosen for a participant. This design 
rormits researchers to estimate the impact of a particular program activity in addition to 
estimating overall program impacts - both of which are of interest to program operators. As is 
often the case, however, factors changed between the time an assignment was made and the 
actual activity commenced. The participant could have changed his or her mind about enrolling 
in a specific program, or a selected activity was not available once the training was scheduled. 
Individuals operating at the field level often were opposed to the inflexibility imposed by a 
once-and-for-all random assignment which was not perceived to be in the best interest of many 
JTPA participants. 

Fourth, the additional burden of identifying a control group (who were excluded from JTPA 
training) may result in delivering services to a harder-to-serve experimental treatment group, 
which in turn would harm the SDA's ability to reach or exceed performance standards.^ Such 
an outcome would resu)t in SDAs' receiving fewer incentive funds at the end of the program 
year, funds which many of them had come to rely upon,^^ 

The small benefits from participation palled in comparison to the uncertain size of implicit costs 
to the vast majority of SDAs asked to participate in the NJE. In spite of an altered research design 
plan and an intense recruiting effort aimed at nearly half of all SDAs, only 7% of the sites 
considered for the experiment ultinnately agreed to participate.^'^ The 17 sites chosen were less 
than the expected 20, with one dropping out almost immediately due to specific problems 
encountered once random assignment of participants began. 

In the end, the 16 sites chosen were similar to non-participating SDA\s with regaM to most 
baseline characteristics, including economic conditions, population mix, administrative 
arrangements, service mix, and program performance. However, most of these 16 sites served 
significantly smaller numbers of clients than the 488 non-participating SDAs and also were 
located in small-to- medium sized cities concentrated in the Midwest, In particular, no SDAs 
from large cities were included in the study 

SDAs chosen in a probabilistic fashion could not be induced to join the experiment and the 
"external validity" of the National Experiment was weakened. The 16 sites are not truly 
representative of all SDAs from a statistical viewpoint; and the findings from the NJE should 
only be generalized to the JTPA system as a whole with extreme caution, 

A further problem occurred in the early phase of the experiment. Originally, the research design 
envisioned assigning all participants to a major program activity such as: classroom training, 
on-the-job training and job search assistance. Random assignment would then be made and 
members of the control group would be denied all available JTPA activities and services. This 
design appeared to be draconian to most SDAs, and a compromise was quickly reached that 
would structure a "minimal" amount of job search assistance (JSA) up-front in the testing and 
assessment process before random assignment was made. In this manner, even members of the 
control group would be guaranteed some JTPA assistance, which would lessen the unease of 
those sites considering participation in the experiment. Nevertheless, because control group 
members were not in fact denied all JTPA services, program impact estimates in the National 
Experiment are biased downward somewhat. 
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Intermediate Phase; Participant Recruitment Activity Assignment and Control Group 
Training, The initial recruitment design recognized the potential difficulty SDAs might 
encounter in recruiting sufficient numbers for overall enrollment goals and special target 
groups, since those assigned to the control group would be excluded from participation during 
the experiment Soon after implementation, the one-to-one ratio of treatment to control group 
assignment was discarded for a two-to-one ratio. This reduced the number of individuals 
recruited and lessened the number of individuals who would be excluded from JTPA activities 
during the 18-month experimental period. The actual recruitment problems differed across the 
16 SDAs, from some meeting their desired target treatment groups with ease, to others who 
finally stopped the random assignment process early or who were permitted to assign 
participants on a higher treatment-to-control group ratio.^^ The problem became especially 
severe in SDAs which experienced labor markets with low unemployment rates where 
interested eligible economically disadvantaged forego JTPA for direct entry to the work force, 
and also in a few SDAs that experienced sudden cut-backs in funding unrelated to the National 
Experiment. 

Rccruitnient of eligible individuals was also affected in the NJE design by purposefully omitting 
individualsinaparticipatingSDA whose enrollment could not be left to the random assignment 
process. Certain OJT employers require a "customized" program for those they refer to JTPA 
for classroom training activities or other JTl^A services, and these employers did not want to 
hire individuals they did not personally refer. Other state programs for persons with 
disabilities, or eligible for welfare, require JTPA eligible individuals to participate as a 
requirement for assistance. Pnese individuals were omitted from the experiment which 
weakens the external validity cf the experiment. This problem, however, was expected to be 
minimal since only a small proportion of SDA participants fell into this special grcup.^^ 

A s( cond group of problems came to characterize the intermediate phabo of the NJE once 
individuals were assigned to a JTPA acHvity. In parHcular, the initial objective was to develop 
separate impact estimates for each of the three major JTPA activities along v/ith the overall JTPA 
impact These major activities include: (1) Occupational Skills Classroom Training (OCT), 
which includes those w^ho are mainly assigned to OCT, but who may also be recommended for 
other "services" (e.g., job search assistance, basic education, etc.), with no more than 20% of their 
training in on-the-job training programs; (2) On-the-job Training (OJT), which includes those 
mainly assigned to an OJT program but who may also be recommended for other "services," 
with no more than 20% of their training in OCT programs; and (3) Other Activities (OA), which 
includes those not assigned mainly to a OCT or OJT program, but rather job search assistance 
(JSA) or other "services" and "custom- tailored" OJT/OCT programs.^^ 

However, there were problems with this approach. First it was not possible to force SDAs to 
offer only one major activity to each participant because nom^ial operating procedures often 
combine a ^^ries of activities or services as part of an "Employment Development Plan" (EDP) 
for a client. In an effort to maintain normal operating procedures in SDAs, the experiment 
identified three categorical groups of JTPA activities. Job Search Assistance could be combined 
with more intensive dassroomand on-the-job training activities, with the major activity defined 
as that which accounted for the majority of total training time. While this combination is 
common to most SDA service delivery, the experiment also permitted classroom and on-the-job 
training to be combined as long as one activity accounted for at least 80%< of the total training 
time. In addition, the utilization of the last "catch-all" categorical group varied dramatically 
across SDAs; for purposes of the study, SDAs negotiated to assign from 15% io 70% of all 
experimental participants to this category. No longer was it possible to estimate the impact of 
JSA in the absence of more intense training, a frequently offered alternative by SDAs. In the 
end, combining major training activities/services across the three categories, while maintaining 
SDA program integrity, eliminated the possibility of estimating enrollee impacts accruing to a 
major activity - which was part of the original random assignment design. 
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The second, and potentially more serious, problem developed as a result of selection into and 
out of an initial activit>' assignment. Quite often individuals assigned to a particular activity, or 
even a combination of activities/services, decide they prefer other activities - or the SDA staff 
suggests activity changes due to changes in the financial environment or changes in the 
availability of specific training curriculum. In either case, the "internal validity" of uniquely 
assigned programs under random assignment is weakened. Unbiased estimates of program 
activity can only be derived for assignment to an activity, rather than participation in an activity. 
While participating staff was expected to hold such changes to a minimum, from 10% to nearly 
20% of adults in the treatment group were enrolled in activities different from those originally 
assigned.^'* 

Thus, the random assignment process across activities, which dominated the early design of the 
NJE, became subject to selectivity biases due to normal JTPA operations. In addition, impact 
estimates of JSA activity without intensive training - a commonly used service alternative in 
most SDAs - could not be estimated. 

Another problem occurred during the activity assignment phase, which involved the amount 
of activities and services received by those in the treatment group who were not officially 
enrolled in JTPA. The actual level of services received by treatmen t group "no-shows" is difficult 
to ascertain. The 1993 18-month interim report states only that minimal service was given to 
one-half of a sample of 307 "non-participating" treatment group members, mostly in the form 
of additional JSA following activity assignment or referrals to OJT employers or classroom 
training providers.^^ In an earlier report, however, significant amounts o^f^all types of JTPA 
activities/services were reportedly given to this treatment group sub-set. In fact, an early 
survey of 192 non-enrollees found that nearly 60% received significant treatment. 

The importance of this issue rests on whether or not program impacts should be calculated for 
assignees or for actual enrollees, since to measure the latter one must divide^ the assignee 
program impact by the ratio of treatment group members who enrolled in JTPA. The smaller 
the proportion of enrollees, the greater the estimated enrollee program impact. Thus if one 
includes those treatment group members who unofficially participated in JTPA as "enrrllees" 
and their actual services are minimal, the estimated enrollee program impact is biased 
downward. The amount of training received by enrollees is smaller (i.e. smaller assignee 
impact) and the proportion of enrollees greater. An upward bias results if the amount of 
activities and services received unofficially is significant and these individuals are excluded 
from the enrollee sub-set. In this case, the amount of JTPA activities and services is greater (i.e. 
greater assignee impact) and the proportion of enrollees is smaller. Apparently, the 
post-random assignment changes in actual activities discussed above, along with the 
uncertainty surrounding the count and significance of treatment received, altered the emphasis 
from enrollee activity impacts in the initial experimental design to assignee activity catep;ory 
impacts in the 1993 NJE interim report. No longer would the experiment's program impact 
reflect the returns to training actually received by participants, but rather the returns to 
individuals initially assigned to grouped activities - in which roughly 40% never enrolled in 
JTPA training and nearly 15% of those enrolled received different training than assigned. 

Finally, many individuals assigned to the control group were referred to other subsidized 
occupational training programs, OJT employers, or job search assistance programs run through 
cmplovment service agencies, community colleges or the like. In spite of recall difficulties, 
which may have affected members of the control group surveyed 18 months after initial contact, 
it appears that a substantial proportion of control group members did, in fact, receive some form 
of non-JTPA II-A subsidized training or employment service. 

The most obvious form of such non-JTPA service is job search assistance (including job 
placement assistance), readily available through local employment service offices and 
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community colleges. As such, estimated program impacts in the NjE include some treatment 
members who received mainly JSA — which could amount to only small increments, if any, in 
this activity over and above those in the control group, 

A similar situation existed in the NJE regarding classroom training for occupational skills. 
Whereas 29,4% of adult women and 17.4% of adult men assigned to the treatment group later 
enrolled in classroom training, control group members responding to the 18- month follow-up 
survey indicated that 20.4% of the adult women and 12.9% of the adult men enrolled in similar 
non-JTPA funded classroom programs. These figures indicate that enrollment rates of those 
assigned to the control group were from 69% (i.e. adult women) to 74% (i.e, adult men) that of 
the enrollment rates of those assigned to the treatment group.'^^ 

Accordingly, all "program impact" estimates in the NJE must be viewed as incremental returns 
to the availability of JTPA training relative to non-JTPA funded training — not relative to an 
absence of government subsidized training. This fact is especially important to an 
experimental program that denies services to the control group. Extra referral efforts by the 
JTPA staff to control group members could be expected if the latter were informed they would 
not be permitted to receive any further JTPA assistance for the next 18 months. To the extent 
this occurred, all JTPA program impact estimates are likely to be biased downwards compared 
to the typical case where little if any recruitment and referral services are given to JTPA 
applicants who, after being contacted, do not enroll in training. 

Follow-up Phase; Internal Validity , Even a perfectly designed random assignment experiment 
cannot yield valid results if those contacted during the follow-up, post-intervention period differ 
systematically from those included in the initial experiment. As before, this discussion focuses 
on adults in the experiment since JTPA's Title II adult program is the subject of this report. 

According to the NJE interim report, 14,905 adults filed a four page Background Information 
Form (BIF), which detailed their work history and personal background characteristics prior to 
assignment. Complex statistical analysis of the randomly assigned treatment and control 
groups found the two to be identical in all observed personal and prior work history 
characteristics.^^ As shown in Table A.l below, 18 months later in-person or telephone 
interviews were completed for a follow-up sub-set of 10,144 adult treatment and control group 
assignees. Thus, over 30% of the full experimental sample had either been omitted because 
insufficient time had elapsed since the initially scheduled assignment date or because 
participants failed to respond to either form of follow-up interview. 

While the NJE interim report does not explicitly show how the analytical samples used to 
estimate program impacts for separate activities were derived, one can arrive at the final sample 
sizes by cross-referring tables in the earlier baseline report against the 18-month NJE interim 
report. The derivation of the analysis sample is shown in Table A.l.^ Of adult men, nearly 
one-third were omitted: 17.8% were excluded from the 18-month follow-up because their initial 
assignment date occurred late during the 1987-1989 experiment period, which precluded them 
from having a full 18-month follow-up period. Another 17.6% did not respond to the interview. 
Similarly, nearly 30% of adult women were omitted from the 18-month follow-up: 18% were 
excluded from the survey and 10.9% failed to respond to the survey. 

This is a sizeable reduction in the experimental population. The possibility of drawing 
misleading conclusions from an analytical sub-sample that does not reflect that of the full 
experiment was not addressed fully and forthrightly in the NJE. However, such a discussion 
would be necessary to dispel concerns as to the internal validity of the study results. 

The analysis sample - used to generate program impact estimates ~ was not compared directly 
witli the full experimental sample. The distributions of the population and the sub-set need to 
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TABLE A.1 



DERIVATION OF ADULT ANALYSIS SAMPLE FROM 
FULL EXPERIMENT BY ACTIVITY AND GENDER GROUPS 



GENDER-ACTIVITY 
SUB-SET: 


FULL 
EXPERIMENT 
SAMPLE 


EXCLUDED FROM 
18 MONTH 
SURVEY 


MON-RESPONDENTS 
TO 18 MONTH 
SURVEY 


IMPACT 
ANALYSIS 
SAMPLE 


[NO] 


[NO] 


(Pcri 


[NOl 


ipcn 


[NO] 


IPCT] 


ADULT MEN: 
















OCT 


1586 


233 


14.7 


296 


18.7 


1057 


66.6 


OJT/JSA 


3182 


428 


13.5 


504 


15.8 


2250 


70.7 


OA 


2080 


561 


27.0 


407 


19.6 


1112 


53.5 


SUB-TOTAL 


6848 


1222 


17.8 


1207 


17.6 


4419 


64.5 


ADULT WOMEN: 
















OCT 


3409 


482 


14.1 


NA 


NA 


(2847) 


NA 


OJT/JSA 


2660 


338 


12.7 


NA 


NA 


(2287) 


NA 


OA 


1988 


630 


31.7 


NA 


NA 


(1340) 


NA 


SUB-TOTAL 


*8057 


1450 


18.0 


882 
(133) 


10.9 
(01.7) 


5725 
(6474) 


71.7 
(80.4) 


















TOTAL 


14905 


2672 


17.9 


2089 


14.0 


10,144 
(10,893) 


68.1 
(73.1) 



* Figures in parentheses refer to the number of cases that were omitted from the 18-month follow-up survey after 549 
non-respondents whose post-assignment UI earnings were used to approximate the foUow-up survey earnings, and thus reduced 
the number of adult women who would otherwise have been excluded from the analysis sample. 

SOURCE: Figures calculated from tables in Bloom (1991) and Bloom et al. (1993). 



be compared with regard to personal demographics, program participation, and 
pre-assignment work histories. In addition, the 18-month interim report pooled data across 16 
separate sites that were not chosen randomly. Since the sites differed greatly, the distribution 
of individuals by sites between the population and the analysis sub-set needs also to be 
discussed. The only comparisons made in the 18~month interim report addressed the response 
bias issue for the two genders between the 18-month survey group (which excluded 18% of the 
population because of late assignment) and the analysis sample (which excluded another 14% 
due to non-response). Non-resp>onse bias was not found in the p>ost-entry earnings analysis for 
adult men, but was found for adult women. The latter was corrected by using follow-up 
Unemployment Insurance earnings data in place of the follow-up survey earnings for those who 
did not respond to the survey instrument.^ 

What is of importance, however, is the extent to which the analysis sample may differ from the 
full experimental sample, which was never addressed in the NJE. Implicit in this oversight is 
the assumption that the first exclusion (due to late assignment during the experiment and 
accounting for 18% of the 30% omitted individuals) results in the same distribution of 
ii .di viduals in the follow-up as that which would occur nmdomly. However, Table A.l suggests 
that this may not have happened, since the final (known) distributions of the analysis sample 
at times appear to differ from that of the full experiment. For example, adult women account 
for a greater proportion of the analysis sample (6,474/10,893, or 59%) than in the full 
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experimental population (8,057/14,905, or 54%). Second, individuals assigned to "other 
service' are under-represented in the analysis sample for both genders. Adult men assigned 
to the less {»^fensive training activity account for only 25% (1,112/4,419) of the analysis sample, 
but 31 % (2,080/6,848) of the full experimental population (the adult women figures are 21% and 
25% respectively). These differences are not meant to imply that significant differences exist 
between the analysis sample and the full JTPA experimental population, but only to indicate 
tha t this issue should be addressed directly in the NJE before the results are taken to be internally 
valid. 



Stimmary 



In summary, the original NJE research design was altered once the experiment was 
superimposed upon existing SDA program operations at the 16 sites that volunteered to 
participate in the random assignment experiment At all three phases of the experiment — site 
selection, program implementation, and follow-up evaluation — program complexities 
introduced potential for impact estimates of the total program and of specific activities to deviate 
from what may have been anticipated in the original design stage. While the extent of these 
potential biases is imknown at present, the biases must be acknowledged before the 
experiment's findings are interpreted as the final answer regarding the effectiveness of JTPA 
for economically disadvantaged adults.^^ 

Even though the 16 sites displayed characteristics common to many SDAs, they were not 
randomly chosen and were not statistically representative of all SDAs. While these criticisms 
do not imply that the 18-month interim report results are meaningless, they do suggest that any 
generalization from this follow-up evaluation should be highly discouraged. 

Further, and to varying degrees, the original research design was compromised during the 
implementation phase in several ways. All participants in the experiment - treatment and 
control group ~ received minimal JTPA services directly. SDA staff's referral efforts gave 
assistance to control group members to help soften the blow of being denied JTPA assistance 
for the next one and one-half years. Furthermore, the ability to estimate program impacts for 
receipt of services by specific major training activities was weakened as some treatment group 
members changed their minds about tne activities to which they had been assigned, and enrolled 
in others or none at all. Over one-third of individuals assigned to the treatment group never 
enrolled, and roughly 15% of those that did enroll participated in activities differing 
substantially from that initially assigned. No information is given regarding the proportion of 
enrollees who completed JTPA. Accordingly, the national experiment altered its initial focus 
from the acquisition of training to the assignment to training. It is little wonder then, that the 
National JTPA Experiment program impacts are relatively low, and often statistically 
insignificant. 

The final categorization of activities combines individuals with varying amounts of training. 
Basic education for high school drop-outs is included with classroom training for occupational 
skills, and JSA is eliminated as a separate category. This basic service component is highly 
characteristic of JTPA service delivery; yet it cannot be analyzed within the final research design. 
In the end, only the total program impact estimates for assignees are put forward for 
determining whether or not JTPA "works."^^ 

It is important to emphasize how one may misinterpret the impact estimates of the NJE. 
Whereas most readers may want to know whether or not federally funded training programs 
improve the earnings and employment of the economically disadvantaged compared to what 
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they would experience without such government assistance, the NJE answers a different 
question. In particular, the reader is given conservative estimates of the earnings and 
employment impacts of recommending a particular JTPA Activity/service compared to 
recommending similar services provided by non-JTPA providers. While this difference in 
"recommended services" is most evident with respect to on-the-job training alternatives (few 
provided outside of JTPA), it is less evident with respect to classroom training, which is widely 
available under non-JTPA funding. In addition, less costly forms of job search assistance 
comprise the most readily available form of non-JTPA employment assistance. 
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Endnotes 



1. A recent study by Couch (1992) undertook an 8-year follow-up of the experimental 
population and found significant earnings impacts each year, varying from $250 to $525. These 
earnings premiums covered program costs resulting in a small, but positive internal rate of 
return. Later earnings gains would increasingly improve the cost-benefit internal rate of return 
from this social experiment. 

2. The majority of infomiation reported in this section has been taken from four early reports 
on the National JTPA Experiment — including: Doolittle and Traeger (1990); Orr, Gueron and 
Bloom (1990); Bloom (1991) ~ and the 18-month interim report, Bloom et al., (1993). 

3. Classical experiments of JTPA using random assignment were developed by USDOL for 
dislocated worker programs during the early 1980s. The range of program impact estimates 
and issues regarding follow-up procedures are summarized in Bowman (1986). 

4. These figures are taken from Bloom et al. (1993), Exhibit S.l, p. XXXVI. It may be noted the 
"earnings impact" reported in the NJE is derived from the post*assignment eaniings of all 
treatment and control group members, including those with zero earnings over the observed 
period. As such, this impact measure combines both an employment effect (i.e. probability if 
employed) and an earnings effect (if employed). These two effects were estimated separately 
in this Utah Title Il-A adult study. 

5. These figures are taken from Bloom, et al. (1993), Exhibit S.6, p. XLV. 

6. In the first 5 months of site selection, none of the top 73 priority sites in the probabilistic 
sample agreed to participate given the original research design (Doolittle and Traeger (1990), p. 
61). 

7. Sites that agreed to participate in the national experiment were paid a negotiated fee for 
higher administration costs related to participation which ranged from $40,000 to $170,000 
(Doolittle and Traeger (1990), p. 51). 

8. Active recruitment by an SO A may also reflect a lack of information of JTPA to the eligible 
population or a lack of quality programming. 

9. Over one-half of SDAs contacted for the study listed the ethical problem of random 
assignment as a major concern (Doolittle and Traeger (1990), p. 37). 

10. Only 25% of those who voluntarily inquire about JTPA services are expected to enroll in 
an activity (Doolittle and Traeger (1990), p.50). For those who must be recruited, the expected 
enrollment rate would be even lower. Another related problem to the random assignment 
process is the potential to delay assignment from the initial point of contact. It is estimated 
approximately one-half of the remaining JT^A applicant pool will be lost for each additional 
visit required during the pre-enrollment period. These delays are often used by SDAs lo "select 
out" less motivated applicants (Doolittle and Traeger (1990), p. 50). 

11. Almost one-half of SDAs contacted for the NJE feared the study could seriously impact 
their ability to meet state-required enrollment goals (Doolittle and Traeger (1990), p. 37). 

12. A side-effect of this policy change was to reduce somewhat the statistical confidence of 
program impacts resulting from a deviation from the optimal one-to-one ratio of treatnicnt to 



85 ''2 



BEST copy /IVAIIABIE 



control group members without a corresponding increase in sample size. See Cave (1987) for 
more details. 

13. Roughly 90% to 95% of those considered eligible for Title II-A are not serviced by JTPA 
(Doolittle and Traeger (1990), p. 38). 

14. Of the 16 SDAs selected for the study, 11 experienced falling unemployment averaging 2 
percentage points, while the remaining 5 experienced constant or slightly rising unemployment 
(Doolittle and Traeger (1990), p. 65). 

15. One-fourth of SDAs contacted for the NJE cited this as a major concern for participation 
(Doolittle and Traeger (1990), p. 34). 

16. Five SDAs requested that adjustments be made to their performance standards due to their 
participation in the experiment, of which 4 were given permission to make adjustments to be 
"held harmless" for program participation (Doolittle and Traeger (1990), p. 70). 

1 7. The research design was changed to ease recruitment efforts in three phases, with the initial 
phase resulting in only 2% acceptance of 48 sites considered. During the next 4 months the 
recruiting :'ffort raised the acceptance rate to 5% for 61 sites considered, finally reaching a 10% 
acceptance rate among the 89 SDAs considered. In all, 229 SDAs were considered with 16 
accepting, 170 rejecting, and 43 being dropped from consideration for various reasons (Doolittle 
and Traeger (1990), p. 92). 

18. A discussion of the similarity and differences found among baseline characteristics are 
found in Doolittle and Traeger (1990), Chapter Five and in Bloom et al. (1991), Chapter Three. 
It may be noted, the largest SDA in Utah, Wasatch-Front South, was heavily recruited for the 
experiment but declined to participate. 

19. Five smaller sites increased the treatment/ control ratio, whereby 51 1 adults were assigned 
under a 3:1 ratio and 202 adults were assigned under a 6:1 ratio. Another site ended random 
assignment early due to a large plant shutdown, which imposed heavy workloads on the ]TP A 
staff that could not be accommodated under the experimental work conditions (Bloom (1991 ),p. 
78). 

20. The national unemployment rate fell from 7.0% to 6.2% bet\\'een the time the experiment 
began and the start of random assignment (Doolittle and Traeger (1990), p. 64). 

21. An additional problem of external validity characterized the research design as eligible 
individuals contacted in the experiment either refused to fill out the four page Basic Information 
Form (BIF) or who filled out the form but were never randomly assigned (Doolittle and Traeger 
(1990), p. 127). The extent of this problem is uncertain, however, as they may have been less 
motivated to participate/work or may have had better employment opportunities than could 
be expected under a chance of being assigned to a JTPA training activity. This problem may be 
small, however, as relatively few failed to complete the BIF. 

22. As noted in the text, the emphasis here is on the major enrollment activity, as all participants 
may be exposed to minimal supportive services and even brief job search assistance. 

23. On average, 75% of JTPA participants enroll in only a single activity or service (Doolittle 
and Traeger (1990), p. 78). 

24. See: Bloom et al. (1993), Exhibit 3.19, p. 69. 

25. Sec: Bloom et al. (1993), Exhibit F.l, p. 372. 
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26. Sec: Orr, Gueron and Bloom (1990) p. 51. 



27. Sec Bloom (1984) for an explanation of this statistical methodology. 

28. These figures are calculated from Bloom et al. (1993), Exhibit 5.7, p. 148 for men and Exhibit 
4.8, p. 93 for women. 

29. See: Bloom (1991), Appendix D. 

30. The figures in the table were derived from internal Abt Associates reports. It may be noted 
the interim report did not indicate the activity assignment of the 882 adult women who failed 
to respond to the 18-month follow-up survey. Of this total, 749 had UI earnings records that 
were used as a proxy for the 18-month follow-up earnings. As such, the 18-month survey of 
6,607 women was reduced by only 133 cases (882 non-respondents less the 749 who h?d UI 
earnings records), resulting in an analytical sample of 6,474 women. 

31 . According to Bloom et al. (1993), Exhibit D.3, p. 342 - program impact estimates for adult 
women were biased upwards by 20% ($645 program impact estimated with UI wages of 
follow-up respondents only as compared with a $539 program impact estimated with UI wages 
of both follow-up respondents and non-respondents). 

It may also be noted the follow-up survey quarterly earnings used in the interim report were 
based upon respondents having to recall their total earnings on a quarterly basis for the 6 
quarters following assignment. No analysis of the comparability between survey-based data 
and the UI earnings data was given in the 18-month interim report. 

32. Social experimentation evaluation methodologies have been criticized in the literature, 
including Cook and Campbell (1979) and Hausman and Wise (1985), and more recently by 
Heckman (1992) and Levitan (1992). 

33. It has been argued that current randomized experiments like the National JTPA 
Experiment do not offer insight into the specific mechanisms by which social programs work, 
and as such only yield a "black box" answer as to whether or not social programs are effective 
in the short-run. (See: Heckman (1992).) 
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■ APPENDIX B: Estimating Selection 
Bias Of Non-Experimental Models 

As discussed in the text, tlie general consensus regarding estimates of non-<^xperimental impacts 
of training programs for the economically disadvantaged has been that they are highly variable 
in size and are unlikely to approximate the "true" value, which can only be known through 
random assignment field experiments.^ Non-experimental net impact studies use comparison 
(rather than control) groups drawn from large-scale external data bases; the comparison groups 
are composed of individuals "similar" with respect to observed demographics and prior work 
jbistories of the targeted treatment groups. They cannot, however, ensure that unobserved 
factors related to program participation are unrelated to later program earnings, which biases 
program impact estimates due to the non-»'andom selection process. The wide variability in 
estimates of program impacts for the economically disadvantaged target groups is thought to 
exemplify the problem of selection bias. 

The only way to "prove" if impact estimates from a non-experimental (treatment versus 
comparison group) study differ significantly from an experimental (treatment versus control 
group) study is to evaluate the outcomes of a single treatment group with both comparison and 
control groups. This rare opportunity existed with the National Supported Work (NSW) study 
v/hich served both high risk youths and long term welfare mothers without young children and 
who lacked prior work experience.^ Following the national demonstration, researchers made 
such comparisons using different comparison groups and different modeling specifications to 
estimate the selection bias of non-experimental program impact models.^ 

Later, a series of statistical tests were developed to identify and eliminate from consideration 
non-experimental program impact estimates, which are likely to biased.'* The findings suggest 
that non-experimental results can duplicate experimental results for those cases which pass a 
series of selection bias tests. To the extent this is true, non-experimental procedures may become 
valid evaluation tools our nation's employment and training programs. 



The Lalonde Study 



In the first study, LaLonde derived alternative comparison groups from random samples of 
adult female household heads taken from the Panel Study of Income Dynamics (PSID) data set 
(595 observations) and the March 1976 Current Population Survey (CPS) data set (11,132 
individuals). Annual earnings from Social Security Administration (SSA) were appended to the 
CPS data set, while internal annual earnings figures were used for the PSID data set. 

A weakness in choosing these samples was the relatively small number of AFDC participants. 
The PSID sample was composed of only 30% AFDC recipients (173 adult women), while the 
CPS sample contained only 2% AFDC recipients (241 adult women). Because there were so few 
welfare recipients, the pre- and post-enrollment earnings of these comparison groups were 
significantly higher than the experimental controls, reaching ten times the level in the 
pre-cnrollment pcri<xls. In addition, while the characteristic pre-enrollment "dip" in earnings 
of economically disadvantaged adults was observed for the treatment group (-26%), a small 
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positive prc-enrollmcnt earnings gain was observed for both comparison groups (+2% and +9% 
for PSID and CPS respectively). 

The result of comparing a highly dissimilar set of comparison groups to the experimental 
treatment group was very dissimilar impact estimates. The program impact from the interview 
surveys of experimentals (treatment versus control groups) was $861 (in 1982 dollars) for 1979. 
The estimated impacts (following a non-experimental design) for even the sub-set of comparison 
group members who were AFDC recipients varied from $1,500 to $2,400 for the PSID sample 
and from not significant (but positive) to $2,000 for the CPS sample. ITius the non-experimental 
design estimates in the study were found to be significantly higher and they varied to a large 
degree from those reported in the experimental design. The reported difference in estimated 
impacts of the experiment and the non-experimental estimates of program impacts were taken 
by the author as proof that non-experimental studies should not be relied upon for policy 
considerations. 

Before accepting these difference in program impacts as evidence that non-experimen tal studies, 
in general, can not be relied upon to yield unbiased estimates, it is important to point out two 
fundamental flaws in the Lalonde study. These flaws, if corrected, couH result in more stable 
and unbiased estimates of non-experimental program impacts. 

First, a response bias in reported post-program earnings was introduced when only 75% of the 
experimental treatment groups responded to the follow-up survey, compared with thejull 
sample of comparison group members whose earnings were taken from SSA records.^ In 
essence, post-program earnings were not observed for 1 in 4 members of the treatment group, 
but earnings were observed for virtually all those in the comparison group. If treatment group 
non-respondents were less likely to be employed or working full time than the respondents, 
then one would expect the program impacts to be biased upwards, as only the "more 
employable" members of the treatment group are used in the statistical analysis. 

Secondly, it may be argued that LaLonde used an inappropriate statistical technique to remove 
the selection bias problem in models which were run on the sub-set of AFDC recipients. 
Following earlier evaluation designs of Comprehensive Employment and Training Act 
programs, he specified the net impact as the change in earnings between 1975 and 1979 rather 
than the level of earnings in 1979. The reason for this specification is that unobserved factors 
which may cause selection bias would be eliminated if they have a constant, or fixed, effect on 
earnings in any pericxi. If one specifies the program impact as the change in earnings, these 
"fixed effects" cancel out. 

To see this more dearly, assume, for example, that adult women who have greater motivation 
to work will earn more money in any given year regardless of program participation, and will 
also be more likely to take part in the supported work study program than women without such 
motivation. If a simple comparison is made of the level of earriings of these two groups, failure 
to account for this unobserved factor (motivation) will bias the program impact upwards. Not 
all of the earnings gain would be due to the program itselt; rather some of the earnings gain 
would be due to the omitted variable, motivation, which is correlated with observed program 
participation in the model. The total impact of program participation in the model measures 
both the indirect impact of the omitted motivation factor (working through participation) as 
well as the direct impact of observed participation. 

By specifying a change in earnings, however, the fixed effect of motivation on each year's level 
of earnings cancels out. (If more highly motivated adults earn $1,000 more each year, then the 
impact of motivation from one year to the next is zero.) As such, the omitted factor, motivation, 
should not be correlated with either program participation or the change in earnings. Selection 
bias would not be a problem in estimates of program impacts. 
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A problem with LaLonde's approach for the AFDC sub-set is that he was forced to use the 
pre-en try year as the base year in his calculations. As noted earlier, economically disadvantaged 
adults typically display a "dip" in earnings during this period, which reflects mainly a transitory 
component (e.g., 'T^ad luck") of their underlying pre-entry earnings process. Following 
enrollment, participants' eaniings reflect both a return to their more permanent trend rate of 
earnings growth and the impact of receipt of training. Thus the impact of the unobserved factor, 
motivation, on the transitory level of earnings in the immediate pre-program period can not be 
assumed to equal its impact on the more permanent level of post-program earnings. By 
specifying the program outcome as the change in earnings for this sub-set of AFDC recipients, 
LaLonde did not remove the selection bias inherent in the model structure. In this case even 
"fixed effects" program impact estimates for the AFDC sub-set would be biased upward.^ 

In addition to the examination of the AFDC sub-set of recipients, LaLonde specified a more 
complex two-step earnings model on the full sample of PSID and CPS comparison group 
members. In the first step, the probability of program participation was modeled and in the 
second step this information was used in a model of post-program earnings. This more complex 
statistical technique identified a negative correlation between unobserved factors related to 
program participation and post-program eamings.^^ Such a strong negative correlation would 
cause the "fixed effects" model to bias program impacts upwards, as adults who are more likely 
to participate are less likely to realize higher permanent earnings. This, along with the 
measurement error (introduced by specifying the base year as the first year prior to entry), 
caused the non-experimental program impact estimates to exceed the "true" values reported in 
his study when using the PSID and CPS comparison groups. In addition, the more complex 
models found the estimates of program impacts to be far less biased than those derived from 
the "fixed effects" models estimated for the smaller AFDC sub-sets. 

In summary, the LaLonde study is relevant to this report for two reasons. First, it is indicative 
of the importance of carefully choosing a comparison group for non-experimental studies. By 
choosing comparison groups that are highly dissimilar to the treatment groups on important 
observed factors, it appears nearly impossible to account for the dissimilarities among 
non-observed, omitted variables which affect program participation and post-program 
earnings. Program impacts estimated with poorly designed comparison groups are likely to be 
characterized by selection bias. 

Secondly, selection bias can be reduced significantly through proper modeling specification. It 
appears that using a two-step procedure to model program participation and post-program 
earnings explicitly is less restrictive, and may reduce selection bias more than by altering the 
post-program earnings variables as is done with "fixed effects" models. This is especially true, 
when the length of employment and earnings records is constrained to the immedia te year prior 
to enrollment. 



The Fraker And Maynard Study 



In 1987 Fraker and Maynard, who had both worked on the NSW evaluation team, undertook 
an independent comparative study of the non-experimental with the experimental program 
impact estimates for the NSW demonstration study. This study is important more for the ways 
in which comparison groups were selected than in the types of analytical models used for 
estimating program impacts in non-experi'r^^ntal studies. 

The authors derived six separate comparison groups from the March 1976 and 1977 CPS data 
sets, resulting in a sample of 1,995 AFDC adult women welfare recipients, of whom 909 had 
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children six years of age or older. The six comparison groups were composed of various sub-sets 
of the full AFDC sample of women; the groups differed due to successive screens on observed 
characteristics to make the comparison group appear more similar to the control group of the 
NSW experimental demonstration. Most comparison group sub-sets were composed of from 
600 to 900 individuals compared to the 800 individuals in the original NSW treatment group. 
SSA annual earnings were then appended to the CPS records for all experimentals and 
comparison group members. 

Using a relatively simple one-step model to estimate program impacts, the authors found 
significantly different results across the six comparison groups. According to the original study, 
in 1979 the program impact was estimated to be $351. (This figure is based on SSA data and is 
slated in current year dollars as opposed to 1982 dollars used in the LaLonde stitdy.) In contrast, 
the non-experimental program impacts estimated by Fraker and Maynard varied from positive, 
but not statistically significant, to $806 across the comparison groups. Cleariy, the choice of 
comparison groups matters. 

The authors also computed program impacts for successive "years" (i.e. 4- quarter segments 
based upon the baseline interview date) following the initial baseline interview. Of interest in 
these findings are the "fixed effects" model estimates as compared to the estimates of the 
experimentals. The estimates of program impacts declined from $1,243 to $349 over each of the 
three successive years following the entry year. The "fixed effects" estimates were roughly $100 
higher each year, declining from $1,330 to $438. In the fourth post-entry year, both estimates 
indicated that there were no longer any meaningful, or statistically significant, program impacts. 

This study shows that improved corrections for selection bias (via "fixed effects" models in this 
case) can yield estimates of program impacts that are relatively close approximations to the 
impacts estimated with experimental designs. One can only wonder what the non-experimental 
impact estimates would have been, had the authors used the two-step selection correction 
technique pioneered by Heckman (which models program participation explicitly) rather than 
the "fixed effects" technique (which only transforms the model's dependent earnings variable 
from the level to the change in pre- to post-enrollment earnings). 



The Heckman And Hotz Study 



Later in 1989, Heckman and Hotz returned to the NSW experiment to apply newly developed 
selection bias tests to determine if properly specified non-experimental models could duplicate 
the experimental net impact results.^ ^ Once again, they were forced to rely on the CPS national 
data base to derive a sample of 18 to 64 year-old AFEXZ recipients with dependents under age 
16. As before, the comparison groups differed from the treatment groups significantly. 
Whereas the pre-entry earnings of the treatment group fell by 35%, the earnings of the 
comparison group rose by 22%. The comparison group was composed of 10 times the 
proportion of whites, were 2 years younger on average, had more dependents, and half were 
not enrolled during the same period as those of the treatment NSW group. In addition, the 
comparison group had greater prior work experience: 4 times the number of weeks worked and 
6 times the number of hours worked per week. 

The diverse members of the comparison group, along with numerous variations of 
non-experimental modeling specifications, resulted in 64 different models. Like the 
non-experimental rrr\ els estimated for CETA in the 1980s, program impact estimates across 
these models varied widely from -$1,400 to +$1,400. 
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Out of all the models fitted, however, 14 were found to pass various selection bias tests 
developed to determine if the earnings processes prior to and following intervention were 
similar — and thus to reduce significantly the chances that the non-experimental estimates were 
biased. As summarized below in Table B.I, the average non-experimental program net impact 
estimates for this sub-set of estimates were valid approximations of the experimental outcomes. 
For 1978, the average non-experimental estimate of $374 was 85% of the experimental estim.ate 
($440), and for 1979 the average non-experimental estimate of $238 v/as 89% of the experimental 
estimate ($267).^^ Heckman and Hotz were able to show that the wide variation of 
non-experimental program impact estimates could be reduced significantly by subjecting 
alternative models to various selection bias tests, and that the resulting non-experimental 
estimates act as good approximations of the random assignment experimental results. 



Table B.l* 



NSW PROGRAM IMPACT ESTIMATES 


1978 


1979 j 


EXPERIMENTAL RESULTS 


+$440 
(142) 


+$267 
(162) 


WEIGHTED AVERAGE OF LCF MODELS THAT PASSED SELECTION BIAS TESTS 


+$374 
(146) 


+$238 
(152) 



* Standard errors of estimale given in parentheses. 



Summary 



In summary, the two studies of the NSW demonstrati3|j|program, which attempted to estimate 
the selection bias inherent in non-experimental studies, are important more for the issues raised 
than the results produced. Even though the non-experimental designs lacked appropriate 
comparison groups and sufficient information regarding pre-enrollment employment and 
earnings histories, they did validate the importance of well designed non-experimental studies 
in order to n-unimize expected selection bias. These design features are important to this Utah 
study since alternative comparison groups and modeling techniques are examined for the 
importance each play» in estimates of program impacts for Utah's economically disadvantaged 
adults. 



Finally, it is important to emphasize that the more recent work of Heckman and Hotz has shown 
it is possible to apply increasingly stringent selection bias tests to non-experimental models to 
help reduce the likelihood of selection bias. With improved pre- and post-intervention earnings 
measures, use of these selection bias tests appears to yield impact estimates that closely 
approximate experimental outcomes. Furthermore, non-experimental evaluations can be 
constructed within a year or two of program completion, as opposed to the field experiments 
that can take an additional 4 to 6 years between initial design and ultimate completion. Because 
of the far greater costs of designing, implementing, operating, and evaluating follow-up surveys 
in field experiments — evaluating existing programs using non-experimental designs with 
readily available administrative data appears to be a cost-effective evaluation alternative for 
states. 
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Endnotes 
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1 . Economctricians classify such estimators as "inconsistcn t" and "biased" because of modeling 
specification errors. That is, program impact estimates will vary from study to study without a 
genera] tendency for the estimates to converge to the "true" program impact, even with large 
sized samples, if the program participation and post-program outcome processes are not 
correctly specified. This may result, for example if the studies: (1) omit important variables, (2) 
incorrectly measure important variables, or (3) fail to account for the inter-relation of the two 
prcKesscs. 

2. The original design of the National JTPA Experiment called for the identification of both 
non-randomly chosen comparison and randomly chosen control groups for the purpose of 
measuring the extent of selection bias in JTPA impact studies. Due to cost constraints, USDOL 
later reduced the amount of resoL'rces dedicated to the derivation of comparison groups which 
could be used to determine if non-experimental procedures could duplicate the experimental 
outcomes. 

3. See: LaLonde (1986) and Fraker and Maynard (1987). 

4. Heckman and Hotz (1989). 

5. It may be noted that in the sHidy, the author only discussed the possibility of response bias 
between the experimental treatment and control group members. This discussion led him to 
conclude that only a small bias should be expected as the difference in response rates between 
the two experimental groups were marginal. 

6. It may be noted this result rests upon the assumption that unobserved personal 
characteristics, say motivation, have a constant effect on pre- and post-intervention earnings. 
This assumption may not hold true if a major intervention service in JTPA is the alteration of 
mdividuals' motivation to find vv^ork and to improve their skills to earn higher wages in jobs 
they find. Specific counseling services, in fact, are directed toward this particular characteristic 
for individuals with weak prior labor force attachment. They may be as important, if not more 
important, than the actual occupational skills acquired in the employment and training 
programs. 

7. See: Bassi (1984) and Ashenfelter and Card (1985). 

8. Ashenfelter (1978) and Ashenfelter and Card (1985) suggest using an earlier period (or series 
of years) as the base year for fixed effects models, as it will more likely represent a more 
permanent component of the underlying pre-entry earnings structure. 

9. In a follov/-on study (1989), Heckman and Hotz subjected this fixed effects model to 
stringent statistical tests and found it to be characterized by selection bias. 

10. These models were pioneered by Heckman (1979), and since have become widely used in 
labor economic literature. In Bamow, Cain, and Goldberger (1980) they were specifically 
identified as being useful for non-experimental program impact analysis. 

1 1 . Tliese tests are an outgrowth of earlier work. See for example: Heckman and Robb (1985). 

12. These weighted averages are compiled for the group of Linear Control Function (LCF) 
models applicable to grouped data. Other fixed effects models were developed, but did not 

9^ 10 U 
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perform as well as the LCF variants. Since individual earnings data was not available, non-linear 
Heckman-type models were not fit for the NSW samph. 

13. A later study by Friedlander and Robbins (1992) applied the LCF model to individual data 
to estimate program impact on the binary outcome measure of employment status (1 = 
employed, 0 = not-employed) for a group of experimental sites for mostly single female AFDC 
parents. Their results were very mixed, showing a wide variation of impact estimates for even 
those models which passed the chosen selection bias tests. It may be noted, however, that the 
authors did not attempt to use either fixed effects models or more complex two-step non-linear 
models that could have been tested given the nature of the data. In addition, the outcome 
measure was binary, and not continuous as is the case for the earnings models developed by 
Heckman and Hotz. 
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■ APPENDIX C: Pre-Entry Earnings 
Models And Selection Bias Tests 
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REGRESSION DESIGNS USED TO TEST 
FOR SELECTION BIAS IN JTPA IMPACT MODELS 





SELECTION BL^S MODEL 


CMT 
#1 


LCF 
#2 


2SLS 
#3 


FEM 
#4 


HSM 
#5 


DEPENDENT VARIABLE: 












PRIOR EARNINGS 


Y 


Y 


Y 


N 


Y 


CHANGE IN PRIOR EARNINGS 


N 


N 


N 


Y 


N 


INDEPENDENT VARIABLE: 












JTPA ENROLLMENT 


Y 


Y 


Y 


N 


Y 


JTPA ENROLLMENT PROBABILITY 


N 


N 


N 


Y 


N 


AGE (& AGE SQUARED) 


Y 


Y 


Y 


Y 


Y 


SCHOOL DROP OUT 


Y 


Y 


Y 


Y 


Y 


BEYOND HIGH SCHOOL 


Y 


Y 


Y 


Y 


Y 


BLACK 


Y 


Y 


Y 


Y 


Y 


HISPANIC 


Y 


Y 


Y 


Y 


Y 


OTHER MINORITY 


Y 


Y 


Y 


Y 


Y 


WiiLrrAKii KiiL^iJriiilN 1 


Y 


Y 


Y 


Y 


Y 


RURAL SDA 


Y 


Y 


Y 


Y 


Y 


INDUSTRY IN 2ND PRIOR YR: 












AGRICULTURE, MINING, & 


Y 


Y 


Y 


N 


Y 


MANUFACTURING, HNANCE, 
INSURANCE & REAL ESTATE 


Y 


Y 


Y 


N 


Y 


WHOLESALE-RETAIL TRADE, 
SERVICES 


Y 


Y 


Y 


N 


Y 


EARNINGS IN 2ND PRIOR YEAR: 












1ST 6 MONTH EARNINGS 


Y 


Y 


Y 


N 


Y 


CHANGE IN 1ST TO 2ND 6 MONTHS 


Y 


Y 


Y 


Y 


Y 



CMT= Cell Matching Technique Model #1. FEM = Fixed Effects Model #4. 

LCF = Linear Control Function Model #2. HSM = Heckman Selection Model #5. 

2SLS = Two Stage Least Squares Model #3. 

Y = Yes, included in model 
N = No, not included inmodeL 



■ APPENDIX D: Quasi-Net Impact 
Employment And Earnings Models 
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■ APPENDIX E: Gross Versus Net 
Impacts Of JTPA Program Activities 

The findings of the Utah net impact study contained in Chapter Five may be compared to those 
of the National Commission for Employment Policy gross impact study.^ While both studies 
report positive impacts from intensive training, the Utah findings are in contrast to the others 
in terms of the relative size of the program impacts. In particular, the Utah findings suggest 
adult women realize a greater impact from on-the-job training (OJT) than from classroom 
training (CT) while adult men realize a greater impact from classroom programs than on-the-job 
training. 

To examine whether these unexpected resultsare attributed to the statistical methodology used 
in this study, or to the uniqueness of the Utah JTPA system and economic environment, the 
results are compared with those derived from "gross impact" models estimated using only 
^^'■ogram enrollees in the Utah data base. The base of comparison in these models is enrollees 
o participated in the less intensive "other activity" programs, which in Utah enrolls nearly 
all individuals in short duration job search assistance. These estimates are derived directly in 
gross impact models. In contrast, they are inferred ~ or derived indirectly ~ in the net impact 
models above by subtracting the net impacts of the more intensive training activities (CT and 
OJT) from the net impact of those enrolled in the "other activities" (OA) category. 

As evident in Table E.l below, the gross impact models tend to over-estimate the relative 
employment and earnings impact differentials (from OA) compared with the more complex net 
impact models used in the study. More importantly, however, the relative impacts between 
classroom and on-the-job training programs are fairly consistent between gross and quasi-net 
impact models in Utah. For example, adult men enrolled in CT realize greater employment and 



TABLE Rl 



ACTIVITY PROGRAM IMPACT DIFFERENTIALS FROM 
NET IMPACT AND GROSS IMPACT MODELS 



GENDER - 
TRAINING GROUP 


PERCENT EMPLOYED 


EARNINGS IF EMPLOYED 


NET IMPACT 


GROSS IMPACT 


NET IMPACT 


GROSS IMPACT 


ADULT WOMEN 










CTVS OA 


+.092 


+.110 


+$1127 


+$1852 


OJT VSOA 


+.141 


+.170 


+$1304 


+$1639 


ADULT MEN 










CT VS OA 


+.016 


+.030 


+$1180 


+$1345 


OJT VSOA 


+.006 


+.020 


+$473 


+$82220 



12^ 

in 



earnings impacts than those in CJT relative to OA. In fact, the order of relative impacts is the 
same in all cases, except for that reported for the relative earnings impacts among adult women. 
That is, the gross impact results support that typically found - classroom training has a greater 
relative earnings impact than OJT for adult women. (That is, the premium to CT over other 
activities is $200 greater than the premium of OJT over other activities.) This is not supported 
by the quasi-net impact models developed in this study, which indicate a greater relative 
earnings impact for adult women in OJT. 

There are two plausible reasons for this outcome. First, in Utah some OJT contracts were 
develof>ed differently during the program year. In particular, individuals who were not placed 
in classroom training and who wanted more than short job search assistance were directed to 
the OJT program. On an experimental basis, some of these individuals were permitted to find 
employers willing to enter into the program on their own, rather than being placed into such 
arrangements by JTPA placement coordinators. The quasi-net impact findings would suggest 
that adult women who were willing to find such contracts on their own may have unobserved 
factors which made them more employable relative to the type of adult men who found OJT 
contracts on their own. ^ 

Second, the economy in Utah during the post-program period in this study (PY 1987) exhibited 
slow growth, with the service sector leading the more sluggish manufacturing sector. Since 
women were over-represented in the former sector relative to men, those obtaining OJT 
contracts in the service sector would have experienced stronger employment and earnings 
growth. 
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1. Geraci and Romero (forthcoming). 

2. This finding in the Utah study is supported by the National JTPA Experiment which found 
a larger significant earnings impact for adult women in OJT during the last 4 quarters of the 
6-quartcr follow-up period. (As indicated in Bloom, et aL (1993), Exhibit S.6, p. XLV, the 
significant estimate of the earnings impact for adult women from OJT totaled +$518 compared 
with +$332 for those from occupational classroom training (OCT) during this period.) 

3. A greater percent of adult women formerly enrolled in OJT were employer* in the second 
year following enlry than those formeriy enrolled in CT, whereas fewer adult men from OJT 
were later employed compared to those from CT. In addition, adult women from OJT who were 
later employed earned only $250 less than those from CT, compared with adult men from OJT 
who earned $1,000 less if later employed. These figures support the notion that a different type 
of adult women selects into Utah's OJT program than the type of adult men who select into OJT. 
Failure to capture these unobserved factors in statistical modeling would result in biased 
program impact estimates. 
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